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INTRODUCTION 


MySQL powers many of the largest high traffic websites on the Internet. 
All of these installations use MySQL replication extensively to provide a 
scalable and highly available database solution.This book is designed for the 
architect and DBA to help with understanding the foundation, features, and 
options for creating a successful and scalable MySQL solution. 

This book describes the native MySQL asynchronous replication that is 
commonly used, detailing the relative strengths and limitations that can 
affect more advanced operations. There are many options available for im- 
proving native MySQL replication, including new features in MySQL 5.5 
and MySQL 5.6 that are described in Chapter 3. The MySQL ecosystem has 
a large number of utilities and tools to support, manage, and enhance 
MySQL replication and help with data integrity. These include OpenArk, 
Percona Toolkit, MySOL Workbench Utilities, and MySOL HA, which are 
discussed in Chapter 5. Each is important to evaluate for the administra- 
tion of the various approaches for creating a complex MySQL topology. 

Also covered are details for understanding and using multi-master rep- 
lication correctly and safely, and implementing MySQL semisynchronous 
replication. Additional products and add-ons are now available to support 
MySQL synchronous replication, automated failover, and more complex 
topologies. A detailed discussion and examples of Galera replication for 
MySQL and Continuent Tungsten Replicator are included in Chapter 6. 

A working knowledge of MySQL replication is a primary skill for the 
MySQL DBA. Detailed in the appendix are the MySQL Sandbox and a vir- 
tualized environment using VirtualBox for running MySQL replication. This 
is an ideal and recommended approach for testing and evaluating the vari- 
ous options, features, and products that comprise a total MySQL solution. 

In recent years there has been great advancement with MySQL replica- 
tion, and this book aims to cover the leading products and features. While 
every attempt has been made to provide the most accurate information, 
commands, options, and operation, some software described is under 
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development and should be carefully considered before use in a produc- 
tion environment. 


Conventions 
All code in this book is shown in a proportional font. For example: 
mysql» SHOW MASTER STATUS\G 
File: log-bin.000006 
Position: 348043830 

Binlog Do DB: 
Binlog Ignore DB: 
1 row in set (0.01 sec) 

SQL statements in code examples are prefixed with mysql> to indicate 
execution with the mysql command line client. This program is included 
with a full MySQL distribution. When using multiple servers with MySQL 
replication examples, the prefixes master» and slave> are used accord- 
ingly for clarification. For example: 
master» SHOW MASTER STATUS; 

Slave»  SLAVE START; 

All SQL statements listed with these prefixes can generally be performed 
in any alternative MySQL client graphical user interface (GUI) tool; however, 
the \G syntax for vertical display output is a mysql command line client- 
specific directive. 

All SQL syntax within text or code examples is in uppercase. For exam- 
ple, the SHOW SLAVE STATUS statement provides important information 
on the state of replication on a slave. This SQL syntax is provided in the 
standard paragraph font. SQL statements in MySQL are case insensitive. 
This syntax is used only to easily distinguish SQL keywords from other 
database objects or variables in statements and is not required when using 
MySQL. 

A specific syntax or value from a code example that is described in gen- 
eral text is provided in a monofont, for example, the Seconds Behind | 
Master value. 

For any Unix/Linux command, this is prefixed with a $ to indicate a shell 
prompt. For example: 


$ mysqladmin extended-status 
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While this book does not describe the use of MySOL for Microsoft operat- 
ing systems, MySQL is supported on this platform, and many of the stan- 
dard MySQL commands detailed will also operate on Microsoft systems. 
The majority of scripts, utilities, and tools used to support MySQL replica- 
tion that are described in this book will not operate natively with Microsoft; 
however, most administration can be performed remotely on a Linux/Unix 
client connecting to a MySQL instance on a Microsoft operating system. 


About MySOL 

The MySQL database server is an open source product released under the 
GPL V2 license. More information about this license can be found at http:// 
www.mysql.com/about/legal/licensing/index.html. The copyright owner 
of MySQL at the time of this publication is Oracle Corporation. Oracle 
Corporation provides product development, commercial licenses for OEM 
providers, and comprehensive subscription services that includes com- 
mercial support and additional product features. 

More information about MySOL can be found at the official MySOL 
website at http://mysql.com and the MySQL developer zone at http://dev 
-mysql.com. 

The current generally available (GA) version of MySQL is version 5.5. This 
book is written to support MySQL versions 5.0 and later, with specific ver- 
sion differences noted when applicable.The current development milestone 
release (DMR) and next version of MySQL is version 5.6. Documented in 
this book are many new MySQL replication features that are in this current 
5.6 development version. These are subject to the Oracle Safe Harbor state- 
ment that is included here for reference. 


Oracle Safe Harbor Statement for MySOL 5.6 Features 
The following is intended to outline our general product direction. It is in- 
tended for information purposes only, and may not be incorporated into 
any contract. It is not a commitment to deliver any material, code, or func- 
tionality, and should not be relied upon in making purchasing decisions. 
The development, release, and timing of any features or functionality de- 
scribed for Oracle's products remains at the sole discretion of Oracle. 
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Open Source Licenses 

Products detailed in this book are covered under various different open 
source licenses and may have different conditions for use. These include 
the following: 


e GPL The GNU General Public License (GPL) may be version 2 or 
version 3. More information can be found at http://www.gnu.org/ 
copyleft/gpl.html. 


e LGPL The GNU Lesser General Public License (LGPL) can be 
found at http://www.gnu.org/copyleft/lesser.html. 


e BSD Information on the Berkeley Software Distribution (BSD) 
license can be found at http://www.linfo.org/bsdlicense.html. The 
New BSD License/Modified BSD license and the Simplified BSD 
License/Free BSD License are variants of this license. 


e Creative Commons (CC) Details of the various different CC 
licenses can be found at http://creativecommons.org/. 


More information about various open source licenses can be found at 
the Open Source Initiative website at http://www.opensource.org/. A de- 
tailed list of the various licenses can also be found at http://www.gnu.org/ 
licenses/license-list.html. 


Common Technical Abbreviations 

It is expected that the reader of this book have a basic understanding of 
SOL and relational databases. The following commonly used abbrevia- 
tions are important and familiar terms when using MySOL and develop- 
ing software with MySQL. 


Relational Database Terms 


RDBMS - Relational Database Management System 
SQL - Structured Query Language 

DBA - Database Administrator 

DDL - Data Definition Language 

DML - Data Manipulation Language 

ACID - Atomicity, Consistency, Isolation, Durability 
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Hardware Terms 


CPU - Central Processing Unit 

RAID - Random Array of Independent Disks 
SSD - Solid State Drive 

I/O - Input/Output 


Software Terms 

SSL - Secure Sockets Layer 

SSH - Secure Shell 

IP - Internet Protocol 

DNS - Domain Name Services 

GNU - GNU's Not Unix! 

BSD - Berkeley Software Distribution 
GPL - GNU Public License 


Common MySQL Terms 

GA - Generally Available 

RC - Release Candidate 

DMR - Development Milestone Release 


Additional terms that the reader may not be familiar with are described 
at the appropriate time. 


Code Examples 
All examples detailed in this book are available for download from the Effec- 
tive MySQL site at http://effectivemysql.com/book/replication-techniques/. 
Code, scripts, and sample data are also available on GitHub at https://github 
.com/effectiveMySOL/ReplicationTechniques. 

A separate text document of all URLs used is also included on the 
website to enable quick access to these references. 


References 

The MySQL Reference Manual at the MySOL developer zone is an invalu- 
able resource. This can be found at http://dev.mysql.com/doc/refman/5.5/ 
en/index.html. Access to manuals for both older and newer MySQL versions 
can be found at http://dev.mysql.com/doc. 


xxii Introduction 


The Planet MySQL website at http://planet.mysql.com provides an 
aggregation of thousands of MySQL bloggers detailing everything about 
MySQL. This is excellent resource for information on MySQL replication 
examples, experiences, and use cases. 





The Five Minute DBA 


MySQL replication has stopped on a slave server with an error message. 
What is the impact of this error on your application users, your scale-out 
architecture, and your backup and recovery strategy? You have to make a 
choice between skipping the SOL statement and correcting your informa- 
tion to enable the SOL statement to succeed successfully. What is the impact 
on data consistency for each choice? What could the downstream effects be? 
As a DBA, you need to use the information from various tools to deter- 
mine if a problem exists, find exactly what has gone wrong, and then fix the 
underlying issue in order to ensure this does not happen again. 
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In this chapter we will be discussing: 
€ Essential information to diagnose a replication issue 
e Options for correcting stopped replication 


e Understanding causes of replication problems 


The 2 a.m. Alert Notification 


You receive an alert that MySQL replication has stopped on your produc- 
tion slave server that is running MySQL 5.5. This rarely happens during 


work hours or at an appropriate time. 


SHOW SLAVE STATUS 


The first action is to look at the current replication status on the applicable 


server. You achieve this with the following SQL statement: 


slave> SHOW SLAVE STATUS\G 
RR KK RRR RRR RRR RRR RR ke e ke RK 1]. 


Slave_IO State: 





LOW X k k k k ck ckckok ck ck RR RRR KK k k ck ke e 


Waiting for master to send event 


Master Host: 10.0.0.48 
Master User: repl 
Master Port: 3306 
Connect Retry: 60 
Master Log File: mysql-bin.001220 
Read Master Log Pos: 3453586 
Relay Log File: relay-10g.003586 
Relay Log Pos: 3452185 
Relay Master Log File: mysql-bin.001220 
Slave IO Running: Yes 
Slave SQL Running: No 
Replicate Do DB: 
Replicate Ignore DB: 
Replicate Do Table: 
Replicate Ignore Table: 
Replicate Wild Do Table: 
Replicate Wild Ignore Table: 
Last Errno: 1062 


Last Error: 


for key 'user id'' on query. Default database: 


Error 'Duplicate entry '42-2011-04-16 00:00:00' 


'book'. Query: 'INSERT INTO 


product comment(product id,user id,comment dt,comment) VALUES 
(20,42,'2011-04-16 00:00:00','I found this very useful with product Y')' 


Skip Counter: 


0 


Exec Master Log Pos: 3452040 
Relay Log Space: 3453930 
Until Condition: None 


Until Log File: 
Until Log Pos: 
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Master SSL Allowed: No 
Master SSL CA File: 
Master SSL CA Path: 
Master SSL Cert: 
Master SSL Cipher: 
Master SSL Key: 
Seconds Behind Master: NULL 
Master SSL Verify Server Cert: No 
Last IO Errno: 0 
Last IO Error: 
Last SQL Errno: 1062 
Last SQL Error: Error 'Duplicate entry '42-2011-04-16 00:00:00' 
for key 'user id'' on query. Default database: 'book'. Query: 'INSERT INTO 
product comment(product id,user id,comment dt,comment) VALUES 
(20,42,'2011-04-16 00:00:00','I found this very useful with product Y')' 
Replicate Ignore Server Ids: 





Master Server Id: 1 
There are several indicators of a replication problem with this output. 


e The SOL thread is not running, as indicated by 
Slave SQL Running-No 


e Replication lag is unknown, indicated by 
Seconds Behind Master=NULL 


e Error information identified in Last. Errnoand Last Error 


At 2 A.M. in the morning, determining what caused this may not be as 
important as rectifying the problem to ensure your data is up to date for 
your application to use the slave server that has the reported error. 


NOTE Columns in this output have not been ordered by importance. The 
convention of the MySQL product has been to add new columns to the end 
of the list—for example, Replicate Ignore Server ids and 
Master Server Idarenew columns for MySQL 5.5. In MySQL 5.1, 
individual error numbers and descriptions were added for the I/O and SQL 
threads at the end of the list rather than with existing error columns. 


TIP Many alerting systems use rules to determine an error condition. In this 
example, an alert was triggered by the rule Seconds Behind Master - 
NULL OR Seconds Behind Master » 30.Incorporating the output of 
the SHOW SLAVE STATUS command in the email alert is one further step 
toward improving the diagnosis. This saves time, and could enable some form 
of action without having to initially connect to the server. 
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Identifying the Problem 


Instead of automatically correcting the problem, it is important to first un- 
derstand why MySQL replication is not running. Was this due to an error, 
or was replication stopped for some other purpose—for example, a backup 
process or software upgrade? The S1ave SQL Running=No indicator is 
not solely the result of an unexpected error. The STOP SLAVE SQL 
THREAD statement produces the same situation. This is one reason why 
an alert on this condition is not always accurate. An alert should be in place 
for Slave SQL Running=No; however, you do not want to be alerted ev- 
ery morning when the backup process stops replication intentionally, but 
only when the backup process fails and replication is not restarted in an 
appropriate amount of time. MySQL replication includes two threads, as 
shown in the earlier SHOW SLAVE STATUS output, and the I/O thread is 
still running with SLAVE IO Running=Yes. It is important to monitor 
both thread states. 


SHOW CREATE TABLE 


The actual error message from the SHOW SLAVE STATUS output in the 
Last Error column shows a duplicate key error occurred. This can be 
confirmed with a review of the structure of the underlying database table, 
and then looking at the current data: 


Slave» SHOW CREATE TABLE product comment NG 

kkkkxkkkxkkkkkkkkkkkkkkkkkkkk Te row kkkkkkkkkkkkkkkkkkkkkkkkkkk 

Table: product_comment 

Create Table: CREATE TABLE `product_comment^ ( 
^comment id^ int(10) unsigned NOT NULL AUTO INCREMENT, 
^product id^ int(10) unsigned NOT NULL, 
^user id^ int(10) unsigned NOT NULL, 
^comment dt^ datetime NOT NULL, 
~comment~ varchar(1000) NOT NULL, 
PRIMARY KEY (^comment id^), 
UNIQUE KEY “user id^ (^user id'^,"comment dt^) 

) ENGINE-InnoDB AUTO INCREMENT-2 DEFAULT CHARSET=latinl 




















As you can see, there is a UNIQUE KEY called user_idon the user_ 
idand comment_dt columns, which corresponds with the index name in 
the error message. MySQL can only produce a duplicate entry message for 
a UNIQUE KEY or the PRIMARY KEY. The value from the Last_ Error 
column can be used to determine the actual values that were being 
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inserted. The following SOL statement can be constructed to verify the 
data causing the error: 


Slave» SELECT * FROM product comment 


-» WHERE user id - 


-» AND comment dt - 
Ckckckckckckckckckckckckckckck ck ck ck ck ck k k kk k*k*k 1. 


42 
'2011-04-16 00:00:00'\G 


LOW **kkckckckckckckckckckckckckckckckckckck kk kk 


comment id: 1 
product id: 10 
user id: 42 
comment dt: 2011-04-16 00:00:00 
comment: The packaging does not state this requires X 


(1 row in set (0.01 sec) 


Indeed, the SQL statement reported by MySQL replication that caused 
the duplicate key violation was correct. Now what? Is this statement some- 
how invalid? Should we ignore it? Is the data that exists in the table incor- 
rect? Should it be deleted? The product idand comment values are actu- 
ally different, indicating this statement is not an identical statement, only 
the unique key constraint columns of user_id and comment Gt are dupli- 
cated. 

At this time there is insufficient information to make an informed deci- 
sion. One option is to review the master MySQL database to look at the 
data for verification: 


master» SELECT * FROM product comment 

-» WHERE user id - 42 

-> AND comment dt = '2011-04-16 00:00:00'\G 
kkkkxkkkkkkkkkkkkkkkkkkkkkkk 1. row BRR KKK ck k kc k kc kck ck ck ckckckckckckckck ck KKK 


comment id: 1 


product id: 10 
user id: 42 
comment dt: 2011-04-16 00:00:00 
comment: The packaging does not state this requires X 


kkkkkkkkkkkkkkkkkkkkkkkkkkk 2., row BRK KKK KKK RK KKK KKK KEK KKK KKK KEK 


comment id: 2 


product id: 20 
user id: 42 
comment dt: 2011-04-16 00:00:00 


I found this very useful when used with product Y 
set (0.01 sec) 


comment : 
(2 rows in 


We find there is a discrepancy in the master and slave data. That was 
unexpected because this violates the defined unique key constraint. Verify- 
ing the table structure on the master database gives us: 


master» SHOW CREATE TABLE product commentVG 


kckckckckckckckckckckckckokckckckckckckck ck k k k kk 1. row BRR KK KKK k ck k kckckckckckckckckckckck KK KK 


Table: product comment 
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Create Table: CREATE TABLE ^product comment" ( 
^comment id^ int(10) unsigned NOT NULL AUTO INCREMENT, 
^product id^ int(10) unsigned NOT NULL, 
^user id^ int(10) unsigned NOT NULL, 
^comment dt^ datetime NOT NULL, 
^comment^ varchar(1000) NOT NULL, 
PRIMARY KEY (^comment id^), 

KEY ^user id^ (“user _id~, comment dt") 
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=latinl 

















A key exists for the user_idand comment_dt columns; however, closer 
inspection shows this is no longer a unique constraint. As the MySQL slave is 
a copy of the master, you may ask how this happened. In MySQL, it is possi- 
ble for a slave to have a different table structure and still operate normally. In 
this example, the difference has subsequently caused an error. There are sev- 
eral techniques and utilities for comparing database objects between MySQL 
instances to identify differences. These are discussed in Chapters 2 and 5. 


Rectifying the Problem 


We can see that the underlying data on the master is different from the 
data on the slave, and that the SOL statement would seem to bring the data 
toward a more consistent state. One option is to simply start the slave and 
see if some bizarre unexplained situation caused MySQL replication to 
stop unexpectedly, and now it will magically work. In this case, because we 
have reviewed the underlying table structure and table constraints, this is 
not going to result in a successful outcome. 


SOL SLAVE SKIP COUNTER 


A common, although normally discouraged, approach is to simply skip over 
the SOL statement and move onto the next statement in the replication 
stream. This would be achieved by running the following SOL statements: 


Slave» SET GLOBAL SQL SLAVE SKIP COUNTER - 1; 
Slave» START SLAVE SQL THREAD; 


Further verification of the replication status with SHOW SLAVE STA- 
TUS confirms that error has been skipped: 


Slave» SHOW SLAVE STATUS\G 


kckckckckckckckckckckckckckckckckckckckck ck ck k k kk 7l. Y OW *kkckckckckckckckckckckckckckckckckck kc kc kc k kk 


Slave IO State: Waiting for master to send event 


Slave IO Running: Yes 
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Slave SQL Running: Yes 


Last Errno: 0 
Last Error: 





Seconds Behind Master: 0 
Master SSL Verify Server Cert: No 
Last IO Errno: 0 
Last IO Error: 
ast SQL Errno: 0 
ast SQL Error: 
Replicate Ignore Server Ids: 
Master Server Id: 1 























CAUTION It is important that you understand why a SQL statement failed 
before executing SQL. SLAVE SKIP COUNTER. This may only result in one 
error being ignored, and the following SQL statements may cause MySQL 





replication to stop again. How many SQL statements do you skip before 
considering if this was a good idea? 


As you can see, the error is now gone, MySQL replication is running, 
and you can return to sleep. This, however, is not the appropriate solution 
to this problem. What has happened is that you have now caused an incon- 
sistency between the data in the master database table and the slave data- 
base table. For example: 


master» SELECT COUNT(*) FROM product comment WHERE user id-42; 


Slave» SELECT COUNT(*) FROM product comment WHERE user id-42; 








MySQL replication is an asynchronous process and does not perform a 
consistency checksum of the underlying data in the table. As long as a SOL 
statement completes without error, replication will report success regard- 
less of the number of rows affected. In Chapter 2 we will discuss these de- 
sign characteristics and the related issues in more detail. 


8 Effective MySQL: Replication Techniques in Depth 


When working with multiple MySQL instances in a large topology, you 
can change the default MySOL prompt as described in the previous 
examples. This was achieved in the mysql] client with: 


mysql> PROMPT slave> 


TIP For more complex topologies, it is advisable to use additional attributes, 
including the host, schema, and user, in the prompt for the MySQL command 
line client. 


Addressing the Underlying Cause 


There are many reasons why MySQL replication may stop with an error. In 
Chapter 2 we will discuss the most common causes and respective solu- 
tions. In this situation, we discovered that the table schema was different 
with both servers. How did this happen? 

We can determine that the table was changed on the master, but not on 
the slave. There is insufficient auditing history to determine who or what 
performed this operation. This issue occurred because a software upgrade 
of the application schema objects on all MySQL databases was not com- 
pleted correctly. The following statements were found to have occurred on 
the master database by reviewing the upgrade procedure: 
master> SET SQL LOG BIN=0; 
master> ALTER TABLE product_comment 

-> DROP INDEX user_id, 
-> ADD INDEX (user id, comment dt); 
master» SET SQL LOG BIN-1; 

Analysis to determine the cause of this problem required more work than 
reviewing MySOL schema and data. It was not possible to determine this 
precise syntax from any MySOL logs. It was necessary to review the busi- 
ness process that occurred and supporting system logs. This confirmed that 
an application upgrade with schema modifications was performed recently. 

From this SOL, the underlying table structure ofthe product comment 
table was modified to remove the uniqueness constraint for the user. id, 
comment dt index.The reason why this was not applied on the slave data- 
base table is the SET SQL LOG BIN=0 statement. This statement disables 
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the MySQL binary log for the ALTER statement. The binary log is the com- 
munication channel used for replication events to a slave. 

There is a perfectly valid reason for this syntax when executing an AL- 
TER TABLE statement in a large replication environment. To maximize 
uptime of MySQL servers and to not block the MySQL slave SQL thread, 
which is a single thread, running ALTER TABLE commands manually on 
each server provides a means to bypass the MySQL replication single- 
thread limitation. 

The management process for this software upgrade process has failed. 
First, not all servers have this modification, and second, slave servers 
should generally be upgraded first. Third, and more importantly, you as the 
on-call DBA may have not been notified of a software upgrade process 
occurring. It is also possible you were notified and this occurred some time 
ago, and only now an error situation has occurred. 


Rectifying the Problem Correctly 


The correct resolution for this problem is to find out why the table struc- 
tures are different. In this situation, that is, applying the software upgrade 
to the slave server before restarting MySQL replication. You could have 
chosen to modify the structure of this table to be consistent with the mas- 
ter; however, understanding that a change happened does not inform you 
of other changes that may have occurred at the same time. This could be a 
process that involves a lot of work and significant downtime, depending on 
the time to execute schema changes. 

The outcome of this problem highlights that having additional checks 
for schema object consistency is needed. MySQL tables, columns, triggers, 
and stored procedures can all have an effect on replication, as shown in 
this example. As described in Effective MySQL: Backup and Recovery (Mc- 
Graw-Hill, 2012), it is important that your regular backup strategy collect 
metadata on objects and checks with previous versions for any detectable 
differences. In Chapter 5 we will discuss a number of tools that can help in 
determining and correcting problems, including this example. 


TIP Itis always important to keep a copy of your database objects for backup 
purposes. This process can also be used to perform additional validation checks 
between servers in your replication topology for any inconsistencies. 
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Understanding Replication Issues 


MySQL replication is an asynchronous operation that works by processing 
completed DDL and DML statements on the master that are recorded in 
the binary log. A MySQL slave can only have one MySQL master server. 
This limitation is subject to change in future MySOL versions, and is 
already possible when using Tungsten Replicator. The processing of binary 
log statements on individual slaves is via a pull process. In most situations, 
MySQL replication works just fine and without incident. 

Some features of MySQL can complicate replication—for example, tem- 
porary table processing. While data on a MySQL slave can be identical to a 
MySQL master, there are also ways the data may differ. You may choose to 
use different storage engines, and these could result in the same data, or 
with the BLACKHOLE storage engine, different data. Different configura- 
tion options for replication can include or exclude specific tables, or act 
differently when handling SOL errors. Chapter 7 will discuss configura- 
tion options in more detail. 

Itisimportant to understand how replication works and how to use rep- 
lication effectively in your environment for your business needs. 


User Security 


As seen by this example, a software upgrade was able to modify the ex- 
pected results of the replication stream with a SOL command. Alterna- 
tively, does your application have sufficient permissions to do the same? 
The classic GRANT ALL ON *.* privilege for an application user is cause 
for great alarm in a number of situations, including the ability to disable 
binary logging as shown here, bypass slave read-only capabilities, and 
modify global system variables that can affect durability and consistency. 
Chapters 2 and 7 will discuss the SUPER privilege in more detail. 


Configuration Options and Variables 


The MySQL reference manual provides a good introduction and back- 
ground on configuration and implementation options at http://dev.mysql 
.com/doc/refman/5.5/en/replication.html. 
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There are several important configuration variables that can directly af- 
fect replication operation. The change of slave exec mode variable from 
the default value of STRICT to IDEPOTENT would alter the result of the 
error reported in this chapter example. In addition to disabling the replica- 
tion stream, rules on the master or the slave can change what SQL state- 
ments are executed. Chapter 7 will provide full details of important MySOL 
configuration options. 


Conclusion 


MySQL replication is the backbone of any production MySQL infrastruc- 
ture. Replication can be used for read scalability, a failover strategy, a backup 
strategy, geographical support, software testing, and many other purposes. 
The flexibility in these choices show that MySQL replication is a technology 
feature that should be carefully understood, managed, and monitored. 

MySQL replication is not without issues in more complicated situations. 
This chapter has indicated one potential issue. In Chapter 2 we will discuss 
more common replication problems, and in further chapters will show 
how to use replication effectively and what additional features, configura- 
tion, and third-party products can be used in providing advanced replica- 
tion techniques to support any complex MySQL topology. 

Examples and links in this chapter are available for download from 
http://EffectiveMySQL.com/book/replication-techniques. 
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Diagnosing Common 
Replication Problems 


MySQL replication has detected an error condition and has stopped. 
While correction is necessary, why did this SOL statement failure occur? 
How would you avoid this situation in the future? What are the implications 
on your data consistency? Understanding replication conditions can help in 
designing preventive measures and monitoring for problem detection and 
management. 
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In this chapter we will be discussing: 
e Detecting replication errors 
e Managing consistency issues 
e Addressing replication lag 


e Ideal monitoring guidelines 


MySQL Replication Architecture Review 


To understand the features and limitations of MySOL replication, it is impor- 
tant to understand the basic mechanics between a MySQL master and slave. 

As outlined in Figure 2-1, the following are the key steps in the successful 
execution of a transaction in a standard asynchronous MySQL replication 
environment. This is not an exhaustive list of all data, memory, and file I/O 
operations performed, rather a high level representation of important steps. 


1. A MySQL transaction is initiated on the master (Point 1). 


2. One or more SQL statements are applied on the master (Point 2). 
The true implementation of the physical result depends on the 
storage engine used. Generally, regardless of storage engine, the 
data change operation is first recorded within the applicable 
memory buffer. For InnoDB, the statement is recorded in the 
InnoDB transaction logs (note that InnoDB data is written to disk by 
a separate background thread). For MyISAM, the operation is 
written directly to the applicable table data file. 


3. Atthe completion of the transaction, the master binary log records 
the result of the DML or DDL statement(s) applied (Point 3). 
MySQL supports varying modes that may record the statement(s) 
or the actual data changes. 


4. A success indicator is returned to the calling client program to 
indicate the completion of the transaction (Point 4). 


5. The slave server detects a change has occurred in the master 
binary log position (Point 5). 


6. The changes are received (i.e., a pull process) by the slave server 
and written to the slave relay log by the slave I/O thread (Point 6). 
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Figure 2-1 MySQL replication workflow 


7. The slave SQL thread reads the new events from relay log (Point 7) 
and applies all statements in the transaction (Point 8). These 
changes may be recorded as a statement to be executed, or as 
a physical row modification. 


8. A success indicator is returned to the slave replication 
management when the transaction completes. 


In summary, SQL transactions are recorded in the master binary log, 
and the change of this log is used as a triggering event for the slave to pull 
the change. Throughout this book we will discuss different features that 
affect and can alter this default asynchronous behavior. 


Interpreting Replication Information 


Knowing what and where to look for replication information is a key com- 
ponent in the toolbox of a DBA; however, interpreting replication informa- 
tion can sometimes be challenging. Understanding the moving parts, how 
they act together, and how to view the state of replication are essential 
steps to administer MySQL. In the following sections we will outline the 
components of replication and how to use various commands and tools to 
make informed decisions about your replicated environment. 
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Binary Logs 

Replicating data with MySOL default replication requires you to enable 
the binary log. This is specified with the --1og-bin- [base name] op- 
tion when starting mysqld or with the MySQL configuration file (e.g., 
my.cnf). The default value for the base name for the binary log is the 
hostname of the server where MySQL is running. Changing the base _ 
name of the binary logs from the default can help you avoid confusion if 
the hostname of the server changes in the future. The actual binary logs 
will include the base name and a sequenced extension. 

MySQL will also create a binary log index file. This will default to the 
location and base name of the binary logs, and with an . index extension. 
The 1og-bin-index configuration option can be used to change this. The 
following code snippet lists these files based on a defined value in the con- 
figuration file that includes a full directory. 


$ MYCNF="/etc/my.cnf" 

$ BINLOG-^grep log-bin ${MYCNF} | cut -d'-' -f2^ 

$ echo ${BINLOG} 

$ DIR-^dirname ${BINLOG}~ 

$ cd ${DIR} 

$ Is -lh * 

$ cat *.index 

/opt/mysql/binlog/mysql-bin 

-rw-rw---- 1 mysql mysql 49M Apr 30 2011 mysql-bin.000001 
-rw-rw---- 1 mysql mysql 1.1G May 16 06:28 mysql-bin.000002 
-rw-rw---- 1 mysql mysql 772M May 29 21:50 mysql-bin.000003 
-rw-rw---- 1 mysql mysql 400M Jun 5 17:29 mysql-bin.000004 
-rw-rw---- 1 mysql mysql 140 May 29 21:52 log-bin.index 


/opt/mysql/binlog/mysql-bin.000001 

/opt/mysql/binlog/mysql-bin.000002 

/opt/mysql/binlog/mysql-bin.000003 

/opt/mysql/binlog/mysql-bin.000004 

TIP It is recommended that you always specify the base name for log-bin. 
The general recommendation is to provide a generic name without including 


details of the specific host. 


The binary log contains all write activity applied to the MySQL server, 
such as INSERT, UPDATE, DELETE, REPLACE, CREATE, ALTER, and 
DROP statements. Read activity is not recorded in the binary log. If you 
want to record all read and write activity, you can enable the general log 
using the general logand general log file options. 


NOTE Historically, the general log was enabled with - - 10g. This variable is 
deprecated as of MySQL 5.1. 
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CAUTION While the general log provides details of all SQL statements 
executed, it is not recommended to enable this in a production environment due 
to the performance impact of logging a large number of statements. 


Binary Log Analysis 
The binary log, as the name suggests, is an internal format. Using the 
mysqlbinlog client utility, you can view statements that are recorded in 
binary logs—for example: 


NOTE The mysqibinlog utility is the recommended tool for reading 
binary logs. 


$ mysqlbinlog ^pwd^/mysql-bin.000001 

# at 107 

#111030 19:32:04 server id 1 end log pos 242 Query thread id-4 
exec time-0 error code-0 

USE book3/*!*/; 

SET TIMESTAMP-1320003124/*1!*/; 

SET GGsession.pseudo thread id-4/*!*/; 

SET @@session.foreign key checks-1, eesession.sgl auto is null-0, 
@@session.unique_checks=1, @@session.autocommit=1/*!*/; 

SET @@session.sql_ mode=0/*!*/; 

SET @@session.auto_increment_increment=1, @@session.auto_increment_offset=1/*!*/; 
/*1NC utf8 *//*1*/; 

SET GGsession.character set client-33,00session.collation connection-33, 
@@session.collation_server=33/*!*/; 

SET GGsession.lc time names-0/*!*/; 

SET @@session.collation_database=DEFAULT/*!*/; 

CREATE TABLE repl test ( ts INT, dt DATETIME) 

[*i*/; 

DELIMITER ; 

4 End of log file 

ROLLBACK /* added by mysqlbinlog */; 

/*150003 SET COMPLETION TYPE-GOLD COMPLETION TYPE*/; 


An alternative way to gather all SOL information on a MySQL server is 
to use the slow query log and set the 1ong query time to 0. Using the 
slow query log in this manner has the added benefit of being easier to 
parse with utilities, including pt - query-digest and mysqldumpslow. 


TIP It is possible to record the slow query information into the mysq1.s1ow log 
table using the log_format configuration option. This can be of benefit if 
enabled for a few seconds to record all SQL statements, which then can be mined 
for additional information, including the read/write ratio and breakdown on a 
per-table basis. 
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The two primary uses for the binary log are for replication and data 
recovery—more specifically, point in time recovery. These uses can be 
combined to assist in creating a clone of a production server with minimal 
downtime. Like other files in MySQL, the position of the file referenced in 
many SHOW commands represents the byte offset of the file itself. For 
example, the filesize can be found with the system tool stat: 


$ stat mysql-bin.000001 
File: 'mysql-bin.000001' 
Size: 1052302 Blocks: 2064 IO Block: 4096 regular file 


NOTE The command line tool stat has a different output depending on its 
origin. The code listing here is partial output from GNU stat, while the BSD 
stat looks more likea 1s -1 output. 


Alternatively, you could run the following: 
$ ls -1 mysql-bin.000001 
-rw-rw---- 1 mysql mysql 1052302 Oct 30 18:47 mysql-bin.000001 

The information about the binary log in MySQL can be found with the 
SHOW MASTER STATUS command: 


$ mysql -uroot -p -e "SHOW MASTER STATUS VNG" 
File: mysql-bin.000001 
Position: 1052302 
Binlog Do DB: 
Binlog Ignore DB: 
Executed Gtid Set: 

The file size in bytes for the mysql-bin.000001 file is 1052302. The 
position, shown in SHOW MASTER STATUS, is also 1052302. In a production 
system, the binary log can be changing constantly. The information here is 
shown on an idle server to demonstrate the verification steps. 

As shown, the binary log names follow a base name.nnnnnn conven- 
tion, where nnnnnn represents a sequential file number. This number is 
incremented when the individual binary log reaches the size as specified 
by max binlog size, the server is restarted, or the FLUSH [BINARY] 
LOGS statement is issued. 

While mysqlbinlog is the most common approach to review binary log 
files, several other options exist. 


e The SHOW BINLOG EVENTS IN 'file' command can read the 
binary log events using an SOL interface. 
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e The MySQL replication listener includes the MySQL Binlog 
API. This can be used to read and process events in the MySOL 
binary log. This is discussed in Chapters 5 and 6. 


€ The various replication pre-fetch tools read and process the binary 
log. These tools can be used and modified to identify information. 
These tools are discussed in Chapter 5. 


e Tungsten Replicator supports many different replication topologies. 
This product first reads and processes the MySOL binary log. This 
open source product can be used and modified appropriately to 
analyze binary logs. This is discussed in Chapter 6. 


The binary log contains a wealth of data that can be mined to provide 
interesting information. The following command analyzes a binary log and 
provides DML statistics broken down by individual table. You can use this 
approach with various mysqlbinlog arguments to also determine statis- 
tics for any time duration. For example: 


$ mysqlbinlog /path/to/mysql-bin.000999 | \ 


grep -i -e "^update" -e "^insert" -e "^delete" -e "^replace" -e "^alter" | \ 
cut -cl-100 | tr '[A-Z]' '[a-z]' | \ 

sed -e "s/\t/ /g;s/\~//g;s8/(.*$//;8/ set .*$//;s/ as .*$//" | N 

sed -e "s/ where .*$//" | sort | uniq -c | sort -nr 


33389 update e acc 
17680 insert into r b 
17680 insert into e rec 
14332 insert into rcv c 
13543 update e rec 
10805 update loc 

3339 insert into r att 
2781 insert into o att 


More information about this simple command can be found at http:// 
ronaldbradford.com/blog/mysql-dml-stats-per-table-2009-09-09/. 


Binary Log Management 

By default, MySOL does not remove old inactive binary logs; however, you 
can enable automatic binary log removal with the expire logs days sys- 
tem variable. The default for expire logs days is 0, which means no au- 
tomatic removal of binary logs. As the system variable suggests, assigning a 
value larger than 0 will remove binary logs older than the value specified, 
where"value" is measured in days. 
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Use caution with expire logs days, as MySQL will remove binary 
logs indiscriminately that are older than expire 1ogs days. There are 
checks to ensure that all of the connected slaves are up to date; however, if 
a slave is broken or not connected to the master, the purge will occur any- 
way. For more information pertaining to binary log options, see Chapter 7. 

Running the PURGE [BINARY|MASTER] LOGS command is a manual 
method to remove binary logs. In a production environment, removing bi- 
nary logs is easy to automate based on your requirements as an alternative 
to using expire logs days. One reason you may need to automate the 
removal of binary logs yourself would be to coincide the purge with a point 
in time backup. There are two variations in which you can purge binary 
logs: by specifying the actual log to purge to, or by specifying a DATETIME 
using a BEFORE variant. The following are two examples for purging binary 
logs. 

Example 1 (Purge to a specified file): 


mysql» PURGE BINARY LOGS TO 'mysql-bin.000005'; 


Here, all of the binary logs prior to mysql-bin.000005 on the server 
would be removed, not including mysq1-bin.000005 itself. 
Example 2 (Purge to a specified date and time): 


mysql» PURGE BINARY LOGS BEFORE '2011-10-31 09:00:00'; 


Here, all binary logs that were closed before the specified date would be 
removed. 


NOTE PURGE MASTER LOGS and PURGE BINARY LOGS are 
synonymous variants. 


TIP Effective MySQL: Backup and Recovery (McGraw-Hill, 2012) covers 
the various options for managing binary logs in a production environment to 
ensure adequate redundancy for a disaster recovery situation. 


SHOW MASTER LOGS 
The SHOW [MASTER|BINARY] LOGS command is used to see all of the 
binary logs and their respective file sizes on the server's host. Again, the 
terms MASTER and BINARY are synonymous variants. 

In the following example you will notice that there are six files with dif- 
ferent sequential log numbers and file sizes: 
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master» SHOW MASTER LOGS; 











+------------------ +----------- + 
Log_name File_size 
+------------------ +----------- + 
mysql-bin.000009 150 
mysql-bin.000010 400567 
mysql-bin.000011 72534 
mysql-bin.000012 506830 
mysql-bin.000013 615407 
mysql-bin.000014 723937 
+------------------ +----------- + 


It is important to mention that in the event a binary log is moved or re- 
moved from the file system directly, MySQL will still show a pointer to that 
file with a zero byte File_size even though the file no longer exists on 
the file system. Moving or removing binary logs from the file system di- 
rectly is not a recommended technique. The following example highlights 
the impact of removing files physically: 


$ rm -f mysql-bin.000009 mysql-bin.000010 mysql-bin.000011 
master> SHOW MASTER LOGS; 











+------------------ +----------- + 
Log_name File_size 
+------------------ +----------- + 
mysql-bin.000009 0 
mysql-bin.000010 0 
mysql-bin.000011 0 
mysql-bin.000012 506830 
mysql-bin.000013 615407 
mysql-bin.000014 723937 
+------------------ +----------- + 


You will see the file sizes for physical binary logs that can no longer be 
found are zero bytes. When you run a subsequent PURGE MASTER LOGS 
command you will see the following warnings: 


master> PURGE MASTER LOGS TO 'mysql-bin.000012'; 

Query OK, 0 rows affected, 3 warnings (0.01 sec) 

master> SHOW WARNINGS\G 

kkkkkkkkkkkkkkkkkkkkkkk |l. row kk kckckck ck ck ck kckckckck ck kckokok 
Level: Warning 

Code: 1612 

Message: Being purged log ./mysql-bin.000009 was not found 
kckckckckckckckckckckckckckckckck ck k k k kk D. row X**kkckckckckckckckckckckckckckck ck KEE 
Level: Warning 
Code: 1612 


Message: Being purged log ./mysql-bin.000010 was not found 
kckckckckckckckckckckckckckckckck ck k kk kk 3., row XX kk kckckckckckckckckckckck kk KKK 








Level: Warning 
Code: 1612 
Message: Being purged log ./mysql-bin.000011 was not found 
3 rows in set (0.00 sec) 
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SHOW MASTER STATUS 
As referenced earlier, you can use the SHOW MASTER STATUS command 
to view information about the current binary log. A user running SHOW 
MASTER STATUS needs the SUPER privilege. The current binary log is 
displayed as File: mysql-bin.000001, as in the following example, 
along with the Position: 107, which is a representation of the byte off- 
set in the corresponding binary log: 
master» SHOW MASTER STATUSNG 
File: mysql-bin.000001 
Position: 107 
Binlog Do DB: 
Binlog Ignore DB: 
Executed Gtid Set: 
There are three other pieces of information shown when running SHOW 
MASTER STATUS. 


e Binlog Do DB is populated by listing databases in the system 
variable --binlog-do-db- [database name (s)]. 


e Binlog Ignore DB is populated by the system variable 
--binlog-ignore-db- [database names].Ifno value exists in 
either setting, all write activity for all databases is being inserted into 
the binary log. 


e Executed Gtid Set, new to MySQL 5.6.5, is used to show the set 
of Global Transaction Identifiers (GTIDs) executed on the master. 


SHOW SLAVE STATUS 


MySQL provides a view of all information pertaining to the status of repli- 
cation with SHOW SLAVE STATUS. 


NOTE The \Gstatement terminator for this statement is important for 
readability; \G will display the output of any SQL statement vertically in the 
mysql client. 


The following is an example of the SHOW SLAVE STATUS output from 
MySQL 5.5, the current GA version: 


slave> SHOW SLAVE STATUS\G 
Slave_IO State: Waiting for master to send event 
Master Host: 10.0.0.1 
Master User: repl 
Master Port: 3306 
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Connect Retry: 
Master Log File: 
Read Master Log Pos: 


Relay Log File: 
Relay Log Pos: 

Relay Master Log File: 
Slave IO Running: 
Slave SQL Running: 
Replicate Do DB: 

Replicate Ignore DB: 
Replicate Do Table: 
Replicate Ignore Table: 
Replicate Wild Do Table: 
Replicate Wild Ignore Table: 
Last Errno: 
Last Error: 
Skip Counter: 

Exec Master Log Pos: 
Relay Log Space: 

Until Condition: 
Until Log File: 

Until Log Pos: 
Master SSL Allowed: 
Master SSL CA File: 
Master SSL CA Path: 
Master SSL Cert: 
Master SSL Cipher: 
Master SSL Key: 

Seconds Behind Master: 

Master SSL Verify Server Cert: 
Last IO Errno: 
Last IO Error: 
jaast SQL Errno: 
jÓst SQL Error: 
Replicate Ignore Server Ids: 
Master Server Id: 
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SHOW SLAVE STATUS displays information about all aspects of repli- 
cation for the given slave host. Reading the output earlier, you will be able 


to see what master server the slave is replicating from. 

The most important components shown in SHOW SLAVE STATUS per- 
tain to files and positions, along with the SQL and I/O slave thread. These 
two threads perform the physical work of writing and reading the relay log 
(the I/O thread) and applying the new events on the slave (the SQL thread). 


e Master Log File The name ofthe current binary log the slave 


I/O thread is currently reading from. 


€ Read Master Log Pos The position in the current master binary 


log the slave I/O thread has read up to. 
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€ Relay Log File The name ofthe current relay log where the 
slave SOL thread is reading from. 


€ Relay Log Position The position of the current relay log read 
and executed by the slave SOL thread. 


€ Relay Master Log File The binary log where the most recently 
executed event by the slave SOL thread resides. 


e Slave IO Running States if the slave server has connected to the 
master host. There are three values: Yes, No, and Connecting (since 
version 5.1.46). Checking the corresponding value of S1ave _ 
running, a system status variable, can also help you determine if the 
slave is connected to the master. 


e Slave SQL Running States if the SOL thread is running. 


There can be, and probably will be, times when your slave host falls 
behind the master given the asynchronous nature of MySQL replication. 
Innovations in MySQL 5.5, including semisynchronous replication and 
features in the 5.6 version of MySQL, have led to many replication im- 
provements. Multithreaded slaves, checksums, and crash-safe slaves are 
just a few of these new features that will continue to be enhanced in future 
releases. These are discussed in detail in Chapter 3. 

Refer to the MySQL documentation at http://dev.mysql.com/doc/refman 
/5.5/en/show-slave-status.html for a full listing of SHOW SLAVE STATUS 
output. It is important you view the documentation for the specific version 
of MySQL in use. 


New MySQL 5.1 Information The following new columns were added in 
MySQL 5.1: 


e Master SSL Verify Server Cert 
e Last IO Errno 

e Last IO Error 

e Last SQL Errno 


e Last SOL Error 
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New MySQL 5.5 Information The following new columns were added in 
MySQL 5.5: 


e Replicate Ignore Server Ids 


e Master Server Id 


New MySQL 5.6 Information The following are new columns in 
MySQL 5.6.5 at the time of this publication and are subject to possible 
change before general availability: 


e Master Info File 

e SQL Delay 

e SOL Remaining Delay 

e Slave SQL Running State 
e Master Retry Count 

e Master Bind 

e Last IO Error Timestamp 
e Last SOL Error Timestamp 
e Master SSL Crl 

e Master SSL Crlpath 

e Retrieved Gtid Set 


e Executed Gtid Set 


Relay Logs 


MySQL uses a numbered set of files on the slave, called relay logs, to hold 
replicated database changes before the SOL thread applies them to the 
slave. The relay log files are numbered in sequence, starting from 000001, 
and are accompanied by what is referred to as the relay index file, which 
contains the names of all relay files currently available. Relay log files are 
in the same format as MySQL binary logs, making them easy to read using 
the mysqlbinlog client utility. 


NOTE The mysqlbinlog utility is the recommended tool for reading the 
relay logs. 
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Like the binary log, a relay log position is a representation of a byte off- 
set in the file, so if the Relay Log Posis671andtheRelay Log Fileis 
mysqld-relay-bin.000002, then MySQL has read up to 671 bytes of 
the corresponding file. The naming conventions for the relay log file can 
be altered with the relay-log-[file name] and the relay log index 
with relay-log-index- [file name] options in the my.cnf file. If 
either of the preceding is absent in the my . cnf file, the relay logs will take 
their naming convention from the pid-file option if specified. 

For example, when the pid-file option is specified in the my. cnf and 
the relay-log and relay-log-index are omitted, the relay logs will be 
mysql 3306-relay-bin.indexandmysqgl 3306-relay-bin.000001. 
If relay-log, relay-log- index, and pid-file are not specified, the 
relay logs will default to host name-relay-bin.nnnnnn and host 
name-relay-bin.index,wherehost nameistheserver host and nnnnnn 
represents the sequential file numbering. 

All events are recorded in the relay log files from the slave I/O thread. 
The slave SOL thread will read these events and execute them on the slave 
host. For example, to examine the relay log using the mysqlbinlog client 
utility: 


$ mysqlbinlog mysqld-relay-bin.000005 

#111030 19:32:04 server id 1 end log pos 242 Query 
thread id-4 exec time-0 error code-0 

USE book3/*!*/; 

SET TIMESTAMP-1320003124/*!*/; 

SET G8session.pseudo thread id-4/*!*/; 

SET GG3session.foreign key checks-1, GGsession.sql auto is null-0, 

@@session.unique_checks=1, 

@@session.autocommit=1/*!*/; 

SET @@session.sql mode=0/*!*/; 

SET @@session.auto_increment_increment=1, 

@@session.auto_increment_offset=1/*!*/; 

/*1NC ut£8 *//*Ix/; 

SET @@session.character_set_client=33,@@session. 

collation connection-33,00session.collation server-33/*!*/; 

SET @@session.lc time names-0/*!*/; 

SET GGsession.collation database-DEFAULT/*!*/; 

CREATE TABLE repl test ( ts INT, dt DATETIME) 

/*1*f; 

DELIMITER ; 

# End of log file 

ROLLBACK /* added by mysqlbinlog */; 

/*150003 SET COMPLETION TYPE-GOLD COMPLETION TYPE*/; 
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Shown in this output is the CREATE TABLE statement that was run 
and described in the output of the master binary log. 


Replication Consistency 


MySQL replication has improved over the years; however, there are still 
issues with data and schema consistency. One of the major concerns with 
replication is the lack of checksums from the master to the slave. There are 
two built-in types of binary logging configurable inside of MySQL. State- 
ment-based replication (SBR) has been a feature of MySQL since 3.23 and 
logs SOL statements to the binary log. Row-based replication (RBR) is rela- 
tively new, implemented in version 5.1, and logs blocks of data instead of 
the statement itself to the binary log. You can manipulate this format with 
the configuration setting binlog format-(ROW|STATEMENT |MIXED]. 
You can also change the binary log format at run-time by using SET 
GLOBAL|SESSION binlog format-(ROW|STATEMENT | MIXED}. There 
are three settings for bàinlog format: 


e ROW Causes logging to be row based. 
e STATEMENT Causes logging to be statement based. 


e MIXED Causes logging to be both row and statement based. In 
some cases, statements are logged in statement format and in others, 
row format. For more information on what statements cause 
statement logging versus row logging, please refer to http://dev.mysql 
.com/doc/refman/5.5/en/binary-log-mixed.html. 


NOTE When changing the binary log format at run-time you will need to 
ensure that any slave host connected to the master has the proper binlog_ 
format enabled before you change the master host. 


NOTE The binlog format for the mysqlbinlog example in the previous 
section was set to STATEMENT. 


In this section we will cover how to identify data integrity issues and 
some examples of their cause. 
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Identifying Data Inconsistencies 


The most effective way to identify data inconsistencies caused by replica- 
tion is to look at your data. There are three types of data inconsistencies 
that need to be identified and fixed: 


e Incorrect data on the slave. 
e The slave is missing a row. 
e The slave has an extra row. 


When using SBR, a slave can easily become out of sync with its master. 
In many cases data inconsistencies are caused by a crash of the mysqld 
process and/or by using nontransactional storage engines, including My- 
ISAM. Given that replication prior to MySOL 5.6 lacked checksum capa- 
bilities, an administrator may never know about inconsistent data from 
master to slave unless they use a third-party tool to check. 

It can be difficult to identify when a slave server has become inconsistent 
with the master. There are, however, tools to aid MySQL database adminis- 
trators in identifying such problems. The widely used Percona Toolkit, origi- 
nally Maatkit, created by Baron Schwartz and now maintained by Percona, 
enables DBAs to identify and fix consistency issues within any given dataset. 
Refer to Chapter 5 for more information about Percona Toolkit. 


Identifying Schema Inconsistencies 


Identifying schema inconsistencies and detecting when schema changes oc- 
cur can be accomplished in a few ways depending on your business needs 
and requirements. 

MySQL has built-in status variables responsible for tracking if an 
ALTER statement was run. The Com alter $ status variables are counters 
that could be tracked and recorded on a specified timetable for each server 
in a MySQL replication topology. As you can see in the following, there has 
been one ALTER TABLE statement issued on this host, given the value of 
Com alter table: 


NOTE The global Com alter $ status variables are reset when mysqld is 
restarted or when FLUSH STATUIS is issued. 
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master» SHOW GLOBAL STATUS LIKE 'Com alter$'; 


Com alter db 0 
Com alter db upgrade 0 
Com alter event 0 
Com alter function 0 
Com alter procedure 0 
Com alter server 0 
Com alter table 1 
Com alter tablespace 0 
Com change master 0 
Com show master status 3 














NOTE All global status variables are reset after a server restart. Including the 
Uptime value with all output can help determine if the variables have been 
reset. 


Schemasync is an open source tool that will identify schema differences 
and create the SOL statements needed to bring a schema up to date. This 
tool can also create the SOL for an undo script just in case you need to re- 
vert the changes. For more information on schemasync, its capabilities, 
and limitations please visit http://schemasync.org. 

MySQL Workbench is a graphical user interface (GUI) tool provided by 
Oracle. One of the features of this tool is schema synchronization. Anyone 
with the proper access and/or a SOL file can run schema diffs from server 
to server or compare a SOL file to a running MySQL server. An adminis- 
trator would also be able to automatically update a workbench model or 
SOL file in either direction; furthermore, workbench models and running 
MySQL servers can automatically update each other in either direction. 

The mysqldump client utility can also be used when comparing schemas 
on different hosts. When mysqldump is used in conjunction with the - -no- 
data option, a diff can be used with the output files. An extension of this 
technique is incorporating md5sum to quickly determine any schema dif- 
ferences between a master and slave(s). For example: 
$ mysqldump -uuser -ppasswd -h[master] --no-data \ 


--Skip-dump-date --databases [db]| md5sum 
45fa52599346a4fab39634b69774301f - 
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$ mysqldump -uuser -ppasswd -h[slave] --no-data \ 
--skip-dump-date --databases [db] | md5sum 
45fa52599346a4fab39634p69774301f - 


NOTE The options - -no-dataand - -skip-dump-date are essential in 
using this mysqldump technique to compare database schemas. 


The problem with the example here is the definition of any primary key 
columns that are auto increment. For an active environment these values 
will change over time. You can address this data change by removing the 
specific syntax from CREATE TABLE statements. Here is an example: 


$ mysqldump -uuser -ppasswd -h[hostname] --no-data \ 
--skip-dump-date --databases [db] | sed -e \ 
"S/AUTO INCREMENT=[0-9]* //" | md5sum 


In the event of a difference in md5sum values you can drill down further 
and check individual tables. The following script performs a more detailed 
table check: 


$ cat pertablemd5.sh 

#!/bin/sh 

SCRIPT NAME-^basename $0^ 

AUTH-"-uroot -ppasswd" 

OPTIONS-"--no-data --skip-lock-tables --skip-dump-date --skip-comments" 
[ $4 -ne 3 ] && echo "USAGE: $(SCRIPT NAME) «master» «slave» <schema>" V 
&& exit 1 

MASTER-$1 

SLAVE-$2 

DATABASE-$3 


[ -z "S(TMPDIR)" ] && TMPDIR-"/tmp" 
mkdir -p ${TMPDIR} 


[ -z “which mysql 2>/dev/null~ ] && echo "ERROR: mysql not found in the PATH" VN 
&& exit 1 


HAVE DIFF-"FALSE" 
for TABLE in “mysql ${AUTH} -h${MASTER} -e \ 
"SELECT table name FROM information schema.tables WHERE table schema " V 


" = 'S(DATABASE]'" |grep -vi "^TABLE NAME" |uniq^ 

do 

mysqldump ${AUTH} -h$(MASTER) ${OPTIONS} --database ${DATABASE} --table STABLE V 
--result-file=${TMPDIR}/master.${DATABASE} .${TABLE}.sql 

MASTER MD5-^cat ${TMPDIR}/master.${DATABASE}.${TABLE}.sql | md5sum^ 

mysqldump ${AUTH} -h$(SLAVE) ${OPTIONS} --database ${DATABASE} --table STABLE V 
--result-file=${TMPDIR}/slave.${DATABASE}.${TABLE}.sql 
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SLAVE MD5-^cat ${TMPDIR}/slave.${DATABASE}.${TABLE}.sql | mds5sum^ 
[ "S(MASTER MD5]" != "${SLAVE_MD5}" ] && \ 

echo "${DATABASE}.${TABLE} is different" && HAVE DIFF-"TRUE" 
done 
[ "S(HAVE DIFF)" = "TRUE" ] && \ 

echo "You can view individual differences with " \ 


"$ diff ${TMPDIR}/*.DATABASE.TABLE.sql" 
exit 0 
This output includes lines that have been modified for display purposes. 
A copy of the script can be obtained from http://EffectiveMySOL.com/ 
book/replication-techniques. 


SBR Schema Validation 

When using SBR, MySQL does not validate a schema difference between 
master and slave. Columns can be out of order and have different type 
characteristics, which can be beneficial, in some temporary cases, but can 
also lead to issues within the database or application itself. In RBR you can 
append columns to the end of a table on a slave; however, column ordinal 
position and datatype validation are enforced. For example: 


master» SET SESSION binlog format-ROW; 
master» CREATE SCHEMA IF NOT EXISTS book3; 
master» USE book3 
master» CREATE TABLE rbr test( 
-> id INT UNSIGNED NOT NULL, 
=> val CHAR(3) NOT NULL, 
-- comment VARCHAR(55) DEFAULT NULL, 
-> PRIMARY KEY (id) 
-> ) ENGINE=InnoDB; 
master> INSERT INTO rbr_test (id,val,comment) 
-> VALUES (1,'aaa','hello test'); 





Now we modify the table on the slave: 


slave> USE book3 
slave> ALTER TABLE rbr_test MODIFY comment TEXT; 


Now executing an INSERT on the master: 


master» SET SESSION binlog format=ROW; 

master> USE book3 

master> INSERT INTO rbr_test (id,val,comment) 
-> VALUES (3,'ccc','hello test'); 
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And finally, on the slave, replication has failed with the following error: 


Slave» SHOW SLAVE STATUS\G 


Last Errno: 1677 
Last Error: Column 2 of table 'book3.rbr test' cannot be 
converted from type 'varchar(55)' to type 'text' 


Causes of Data Inconsistency 


Data inconsistency can be caused by a number of factors. The most com- 
mon reason is human intervention. Other common reasons for inconsis- 
tent data can be attributed to configuration settings, a bug, feature, or crash 
within mysqld itself. 

Deactivating the binary log within a session can be extremely useful but 
potentially dangerous. Changing the session variable SQL LOG BIN with 
SET SESSION SQL LOG BIN = 0 may precede an ALTER table statement 
in a planned schema change; however, if not reenabled within the same 
session, all future statements are not replicated. Automation is the key to 
avoiding a human error event like the one described earlier. The only time 
an administrator should use this technique is when a schema change 
would take too long for a normal maintenance window. 

Setting the system variable, read only,to 1 or ON is highly recommend- 
ed to ensure no data modification can occur on a slave. Unfortunately, if a 
user has been granted the SUPER privilege (which is also enabled with 
GRANT ALL ON *.*), this user can bypass the read only setting on the 
slave. This is an avoidable problem that is solved by using discretion when 
creating application user privileges. 

Running replication while skipping databases and/or tables with 
--replicate-ignore-db or --replicate-ignore-table can also 
prove to be a problem. Session data, for example, stored in a database does 
not generally require replication in most cases; however, if you are ignoring 
the database or table and need to fail over or rebuild a node, you introduce 
additional complications. RBR has improved replicated data consistency 
issues and will be covered in detail in Chapter 3. 
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Common Replication Errors 


There are many common replication errors that occur on slave servers. 


MySQL Server ID 


If, for instance, an administrator set up a new slave manually, the following 
1593 error may occur: 


Slave» SHOW SLAVE STATUS\G 


Slave_IO Running: No 
Slave SQL Running: Yes 


Last IO Errno: 1593 
Last_IO Error: Fatal error: The slave I/O thread stops 
because master and slave have equal MySQL server ids; these ids must be 
different for replication to work (or the --replicate-same-server-id option 
must be used on slave but this does not always make sense; please check the 
manual before using it). 


A 1593 error is when both the master and slave have equal MySQL serv- 
er IDs, and is easy to run into without proper automation and provisioning. 
Fortunately, this is also a very easy problem to fix by simply changing the 
server_id system variable on the slave server to be unique within the 
cluster. Notice that this error stops the I/O thread from connecting to the 
master server. 


Missing Schema Objects 


When running into schema differences, you may notice the 1146 replica- 
tion error. Regardless of how the schema became different, there is a sim- 
ple fix for the example shown here: 


Slave» SHOW SLAVE STATUS\G 


Slave_IO State: Waiting for master to send event 
Slave_IO Running: Yes 
Slave SOL Running: No 


Last Errno: 1146 

Last Error: Error 'Table 'book3.no tbl on slave' doesn't 
exist' on query. Default database: 'book3'. Query: 'INSERT INTO 
no tbl on slave(id) VALUES (1) ' 


Last SQL Errno: 1146 
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Last SOL Error: Error 'Table 'book3.no tbl on slave' doesn't 
exist' on query. Default database: 'book3'. Query: 'INSERT INTO 
no tbl on slave(id) VALUES (1)' 


Fixing this error is as simple as creating the table on the slave host and 
starting the slave. Keep in mind that although the specified fix works, there 
could be records in the table that may not be present on the slave by simply 
adding the table. An additional check of the number of rows on the master 
is more complex, as there may be subsequent SQL statements that create 
this data. In the event of this error steps should be undertaken to under- 
stand why this occurred. 


Ignoring Duplicate Rows 


An observed practice that is not recommended is to skip duplicate key errors 
by default when setting up a slave using the --slave-skip-errors=1062 
configuration variable. In a production system this can lead to data drift 
from master to slave. In this example you will see the impact of enabling the 
skipping of duplicate replication errors: 

master» USE book3 

master» CREATE TABLE uniq test ( 


id INT UNSIGNED NOT NULL 
) ENGINE-InnoDB; 


On the slave we modify the table structure: 


slave» USE book3 
Slave» ALTER TABLE uniq test ADD UNIQUE KEY (id); 


Insert data on the master: 


master» USE book3 

master» INSERT INTO uniq _test(id) VALUES (1); 

master» INSERT INTO uniq test(id) VALUES(1),(2),(3); 
master» SELECT * FROM uniq test; 








4 rows in set (0.00 sec) 
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Review data and replication status on the slave: 


slave» SELECT * FROM uniq test; 








1 row in set (0.00 sec) 
Slave» SHOW SLAVE STATUS\G 


Slave IO Running: Yes 
Slave SQL Running: No 
While there is no replication error, the data is now inconsistent. This 
represents a silent replication failure leading to an inconsistent slave host. 
Under normal operations, without the --slave-skip-errors=1062 
option, the following replication error would occur: 


Slave» SHOW SLAVE STATUS\G 


Slave_IO Running: Yes 
Slave SOL Running: No 


Last_Errno: 1062 

Last_Error: Error 'Duplicate entry '1' for key 
'id' on query. Default database: 'book3'. Query: 'INSERT 
INTO uniq test(id) VALUES(1), (2),(3)' 





Seconds Behind Master: NULL 


Last SQL Errno: 1062 

Last SQL Error: Error 'Duplicate entry '1' for key 
'id' on query. Default database: 'book3'. Query: 'INSERT 
INTO uniq test(id) VALUES (1), (2), (3)' 








In this section we covered just three of the most common replication er- 
rors. There are many more errors and scenarios where replication may 
produce an error. There are some very specific replication errors that can 
occur when using RBR. RBR will be covered in greater detail in Chapter 3. 


Understanding Replication Lag 


Replication lag can occur for many different reasons in many different sce- 
narios. Finding the root cause may not always be apparent; however, in this 
section we will cover the primary reasons for replication lag along with 
other system aspects to check when diagnosing lag. 
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Primary Causes of Lag 


Replication lag primarily occurs due to the asynchronous design and sin- 
gle-threaded application of replication events. Starting in MySQL 5.5, the 
implementation of semisynchronous replication is one step toward more 
replication capabilities. Replication lag has been further improved in 
MySQL 5.6 with multithreaded slave capabilities using the slave 
parallel workers configuration option. More information can be found at 
http://dev.mysql.com/doc/refman/5.6/en/replication-options-slave 
-html#sysvar_slave_parallel_workers. 

Running a backup or an upgrade is another reason to stop replication, 
which will also cause replication lag. To obtain a consistent mysqldump 
backup on a slave instance, a DBA will typically run STOP SLAVE SQL_ 
THREAD before running mysqldump in order to obtain the correct corre- 
sponding master binary log and byte offset/position found in SHOW SLAVE 
STATUS. 


NOTE In MySQL 5.5 it is possible to obtain the master binary log position on 
the slave in mysqldump with the --slave-data option. 


Stopping the slave during a backup is not always the best solution, espe- 
cially if this is used as a read server or for failover purposes. 

There are different replication errors that cause either the SQL or I/O 
slave thread to stop. Running a CHANGE MASTER TO statement and assign- 
ing the incorrect master credentials can cause a simple I/O thread error. 
Replication errors can be as complex as trying to solve a split-brain, pri- 
mary key issue using multiple masters in a complex replication topology. 
Given all of the causes of replication lag, you should consider the following 
when starting to troubleshoot: 


e Replication stopped intentionally (e.g., backup/upgrade) 

e Replication stopped from an error (e.g., slave I/O or SQL thread or both) 
e Master load 

e Slave load 

e Network latency and connectivity issues 


After you have considered what the problem could be, the next step is to 
check the MySQL error log. 
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The MySQL Error Log 


All of the errors covered in the previous section can also be seen and mon- 
itored in the MySQL error log. For example, if you are monitoring the error 
log, and you should be, you would see the following entries in your error 
log when a 1062 error occurs: 


120602 5:56:44 [ERROR] Slave SOL: Could not execute Write rows event on 
table book3.uniq test; Duplicate entry '1' for key 'id', Error code: 1062; 
handler error HA ERR FOUND DUPP KEY; the event's master log binary- 

logs.000003, end log pos 1065, Error code: 1062 

120602 5:56:44 [Warning] Slave: Duplicate entry '1' for key 'id' 

Error code: 1062 

120602 5:56:44 [ERROR] Error running query, slave SQL thread aborted. Fix 

the problem, and restart the slave SQL thread with "SLAVE START". We stopped 
at log 'binary-logs.000003' position 905 





If you were to run STOP SLAVE, mysqld would also record this in the 
error log. For example: 


120611 13:39:42 [Note] Error reading relay log event: slave SQL thread was 
killed 

120611 13:39:42 [ERROR] Error reading packet from server: Lost connection 
to MySQL server during query ( server errno-2013) 

120611 13:39:42 [Note] Slave I/O thread killed while reading event 

120611 13:39:42 [Note] Slave I/O thread exiting, read up to log 
'mysql-bin.000003', position 107 


The same behavior occurs when starting the slave with START SLAVE: 


120611 13:41:36 [Note] Slave SQL thread initialized, starting replication 
in log 'mysql-bin.000003' at position 107, relay log 
'./relay-bin.000008' position: 253 

120611 13:41:36 [Note] Slave I/O thread: connected to master, 

replication started in log 'mysql-bin.000003' at position 107 





TIP The error log is one of the best sources of information you can use to gain 
insight on what might be happening or has happened on a seroer. It is usually 
the first place an administrator should look. 


Simple Techniques to Improve and Minimize Lag 


The simplest way to improve replication lag is to improve SOL performance 
that blocks the single-threaded SOL execution on the slave. Reducing the 
execution time of UPDATE and DELETE statements can greatly improve 
replication lag. Effective MySQL: Optimizing SOL Statements (McGraw- 
Hill, 2011) provides many examples for improving SOL performance in- 
cluding techniques to reduce the number of, and complexity of, 
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SQL statements that are executed. Improving the schema design to remove 
excessive and duplicate indexes will improve the performance of applying 
DML statements on a slave. 

In some cases you may not need to replicate certain databases or tables. 
Excluding certain databases and/or tables will reduce SOL execution on 
the slave host. MySOL replication has the ability to ignore specific data- 
bases and/or tables through the system variables --replicate-ignore- 
db= [database name] and --replicate-ignore-table- [table 
name], respectively. The logic behind this is that if you do not replicate 
certain databases or tables, there will be less relay log processing, and 
there will be fewer writes running to the slave host. That said, recovery of 
these databases or tables will have to considered in detail as to not affect 
production traffic in the event of a failover. 

You should avoid using the - -replicate-ignore-db option if you 
are using cross-database updates. For example, if the slave host was started 
with --replicate-ignore-db=app and you run the following SOL, the 
statement will be replicated to the slave: 
master» USE book3; 
master» UPDATE app.test tbl 

-» SET vall - 5 WHERE vall - 1; 

The update to the table succeeds on the master and is replicated be- 
cause --replicate-ignore-db only applies to the default database 
set by the preceding USE statement. The solution is to use --repli- 
cate-wild* options. 


NOTE The --replicate-* options do not accept multiple values in one 
line. To specify multiple databases or tables, add multiple --replicate-*- 
db or --replicate-*-table options to your my. cnf file. 


Choosing an alternate filesystem type—for example, ext 4 or xfs—may 
provide additional benefits for your production workload. Tuning the filesys- 
tem configuration appropriately with noatime and nobarrier attributes 
can also assist in disk I/O performance. Appropriate hardware is also an es- 
sential part of any database architecture. If the system you are working with 
has a heavy write workload or a mixed read/write workload, consider using 
faster disks, more CPUs, and always more memory. Using the best RAID con- 
figuration and considering SSDs are discussed in the following section. 
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Your MySQL topology is only as good as the weakest link in your infra- 
structure. If you have well-configured hardware running your master, your 
slave also requires appropriate hardware choices. Recognizing the weakest 
link can lead to better system design in the future and higher availability in 
the event of an emergency. For all MySQL slave replication configuration 
options, visit http://dev.mysql.com/doc/refman/5.5/en/replication-options- 
slave.html. 


Advanced Techniques 
to Improve and Minimize Lag 


MySQL can be configured to behave differently in replication by changing 
certain system variables: 


CAUTION Altering the MySQL configuration on a slave may impact your 
production failover or backup strategy. Use caution when altering the 
configuration between servers in any MySQL replication topology, now and 
in the future. At a later time, the requirements and the purpose for the slave 
may alter. 


€ innodb flush log at trx commit By default, the value of 
innodb flush log at trx commit is 1. This means that the log 
buffer is written after every commit and a flush disk operation is 








performed on the log file. You would normally change the default 
setting of innodb flush log at trx commit when your system 





has an I/O bottleneck. Setting the value of innodb flush log at 





trx commit to 2 will write every transaction to the log buffer; 
however, the flush to disk is only a loose interval of once per second. 
Although the default value of 1 is required for ACID compliance, it 
would be easy to automate and recognize when a slave may be 
lagging behind and change this global dynamic value from 1 to 2 
during times of lag. 





CAUTION Setting innodb flush log at trx commit to 1 will 
ensure full ACID compliance. However, setting this variable to 1 also will 
hinder replication performance. For more information please see, http://dev 
mysql.com/doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_flush_ 
log_at_trx_commit. 
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e sync binlog The default value of 0 means that mysqld does not 
control synchronization to disk, but leaves synchronization up to the 
operating system. The system variable, Sync binlog,is another 
server tunable used to speed up replication when there is an I/O 
bottleneck on the system. When the value of sync binlog is set to 
1, the safest setting, events sync to the binary log after every commit, 
which provides, at most, one transaction lost in the event of a 
mysqld crash if autocommit is enabled. Setting sync binlogtoa 
value greater than the default, 0, allows MySQL to sync events at a 
much slower rate (allowing the disk to not work as much). Although 
setting sync binlog to 1 is the slowest setting, it can also be sped 
up with the use of a battery-backed write cache. 


In the event replication falls behind the master, an administrator can set 
sync binlog to 30 and innodb flush log at trx commit to 2 to 





help assist with replication catch-up. The action of changing these two vari- 
ables may cause different problems in a disaster recovery situation. 

You may also gain performance if you store your data and binary logs on 
separate storage partitions. This may affect your backup and recovery 
strategy if you use this slave for that purpose. 

Using the correct RAID configuration within your hardware is another 
way to ensure minimal replication lag. RAID 1 + 0, or RAID 10, is the best 
configuration for database write throughput, whereas the more common 
RAID 5 is optimized for read workloads. Although more disk space is used 
when using RAID 10, the gain in write throughput can prove to be worth it. 
In a RAID 10 configuration, which requires a minimum of four disks, the 
addition of two disks can also make a significant improvement. Popular 1U 
servers generally support the installation of six drives. 

If the writes of the system cause normal spinning disks to run with too 
much I/O wait, you may want to consider running SSD or a flash disk solution. 

Prefetching data on the slave host is another possible optimization tech- 
nique that is designed to read the relay log slightly ahead of the SOL 
thread. The SOL statement is then converted into a SELECT statement, and 
executed in a different thread. This has the benefit of performing the disk 
I/O ofthe read independently, and will result in increasing the speed of the 
write when it is executed. Chapter 5 will discuss this advanced feature in 
more detail. 


Diagnosing Common Replication Problems 41 


Upgrading the version of MySQL can also provide benefits. There are 
new features in MySQL 5.6 that provide performance improvements for 
reducing lag. The binlog group commit (BGC) addition, discussed in 
Chapter 3, is one feature for considering the upgrade of your MySQL ver- 
sion in a high volume environment. MySQL replication can also support 
using a newer version on a slave, i.e, a MySQL master may be running 
MySQL 5.1 or MySQL 5.5, and a slave can be using MySQL 5.6. Some new 
features may be limited as they require MySQL 5.6 on the master. 

Multiple companies are working to improve on the native MySQL rep- 
lication implementation. While many products require a custom MySQL 
binary to be installed and configured, Tungsten Replicator can be installed 
and configured with an existing MySOL topology, including multiple 
MySQL versions, and can be used to take over native replication to im- 
prove replication performance and correct lag. Tungsten Replicator will be 
covered in more detail in Chapter 6. 

It may be time to shard your environment. Sharding is a way to segment 
data into multiple vertical silos or partitions. Typically, sharding is needed 
to scale write activity in an environment, meaning expanding the number 
of master servers you have vertically. If an environment is made up of a mas- 
ter/slave topology where replication lag is prevalent, you may need to 
move some of the operations to a new master/slave topology. From a data 
and schema perspective, this is relatively easy; however, the application 
accessing this data may need to be modified to find the appropriate data in 
multiple locations. 


Monitoring Replication 


It has been stated that Seconds Behind Master in the output of 
SHOW SLAVE STATUS is poorly named, given the implication of its name. 
Seconds Behind Master really measures the latency in seconds the 
slave SQL thread is in relation to the slave I/O thread, or the quantity of 
relay log events not applied on the slave in relation to committed events on 
the master. If you see values higher than 0, you may be dealing with net- 
work latency or high server loads. 

If you want a more accurate measurement of Seconds Behind Master, 
consider running your own heartbeat through the master from the slave 
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server. A simple example can be found at http://datacharmer.blogspot 
.com/2006/04/measuring-replication-speed.html. The Percona ‘Toolkit 
heartbeat utility pt-heatbeat is another example available at http:// 
www.percona.com/. 

SHOW SLAVE STATUS shows a great deal of information needed for 
replication monitoring and will be covered more in depth in Chapter 8. 
Another consideration is disk space utilization, both of the master and 
slave hosts. By default, the size of a binary log is ~1GB, and if you do not 
have a mechanism to remove older files, it could potentially fill up the en- 
tire disk. Chapter 5 provides some utilities to assist with disk management 
of binary logs. 


Conclusion 


Knowing some of the common issues you may find in a replicated environ- 
ment will ensure you are better prepared to administer MySQL. Having a 
solid understanding of the moving parts in replication will help you diagnose 
issues faster when they arise. Every replication installation has different load 
and data manipulation characteristics. It is essential to benchmark and test 
different replication configurations to fit your unique business requirements. 
Armed with the information in this chapter you will be able to avoid common 
replication issues and hopefully avoid potential data disasters. 

Examples and links in this chapter are available for download from 
http://EffectiveMySQL.com/book/replication-techniques. 





Improving Standard 
Replication Features 


In the past, improving replication involved implementing different replica- 
tion topologies and increasing throughput with faster servers and networks. 
While these methods are still important today, MySQL replication is be- 
coming more feature rich, and third party software products have been de- 
veloped to break the asynchronous replication mold. Data drift and single 
threaded asynchronous replication are two of the important issues with 
the current implementation of MySQL replication, but there is hope for the 
future on multiple fronts. 
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In this chapter we will be discussing: 
e Improving on asynchronous replication 
e Securing MySQL replication 
e New replication features in MySQL 5.6 


Extending Asynchronous Behavior 


A common complaint with traditional MySQL replication is that it is asyn- 
chronous. This means that a write operation has to complete on the master 
host before it is replicated and applied to the slave host. What if something 
happens to the data written on the master before this is replicated to the 
slave? There are many new capabilities, both in and outside the MySOL 
server that provide better data integrity and synchronization. 


Semisynchronous Replication 


In addition to the performance improvements in MySOL 5.5, one major 
new feature is semisynchronous replication. This new replication method is 
an intermediate mechanism when compared to asynchronous replication 
and synchronous replication. You can think of semisynchronous replication 
as a step in the right direction toward full synchronous replication. 

Asynchronous replication in MySQL means that events are written to 
the binary log on the master host, and the slave I/O thread independently 
records the event on the slave. There is no guarantee to the master that 
events reach, or have been committed, to the slave host. 

On the other side of the replication spectrum is synchronous replication 
where all transactions are acknowledged and committed on all slaves be- 
fore returning to the session that initiated the transaction. Semisynchro- 
nous, in this case, means that after a commit on the master host, a wait will 
occur until one of the configured slave hosts has logged the event to disk. 
There is a drawback implied here, but we will cover this in more detail 
later in this section. 


Semisynchronous Plugin Installation 
Semisynchronous replication is installed using plugins on MySQL 5.5 and 
higher. This type of replication can be used as an alternative to traditional 
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asynchronous replication. A best practice is to load the semisynchronous 
plugin with the INSTALL PLUGIN command on the master and slave hosts. 
Ensuring the MySQL server supports dynamic loading on both the master 
and slave host is a requirement if you want to install semisynchronous rep- 
lication with these SOL statements. To see if dynamic loading is enabled on 
your MySQL master and slave server, run the following command: 


master» SHOW VARIABLES LIKE 'have dynamic loading'; 











NOTE To install semisynchronous replication dynamically, both the master and 
slave hosts require the system variable have dynamic loading to be YES. 


There are two separate plugins used for semisynchronous replication, 
one for the master host and one for the slave host. 


NOTE You should use the root account, or at least a user that has SUPER 
privilege and INSERT privilege on the mysql . plugin table when installing 
these plugins. 


On the master host: 


master» INSTALL PLUGIN rpl semi sync master SONAME 'semisync_master.so'; 
master» SHOW PLUGINSNG 


KKK KKK KKK KKK KKK KEK ck ck k k kk kk 21. row "XX kk k kk kk ck kk KKK KKK KKK KEK 
Name: rpl semi sync master 

Status: ACTIVE 
Type: REPLICATION 

Library: semisync master.so 

License: GPL 


On the slave host: 


Slave» INSTALL PLUGIN rpl semi sync slave SONAME 'semisync slave.so'; 
Slave» SHOW PLUGINS\G 





kckckckckckckckckckckckckck ck ck ck ck kc kc kc ck k k k kk 21. row kk kk kk kk kk kk kk kk kc kc kc kckckokok 
Name: rpl semi sync slave 
Status: ACTIVE 
Type: REPLICATION 
Library: semisync slave.so 
License: GPL 
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The system variables that toggle the use of semisynchronous replication 
are similar on the master and slave host. A zero (0) value for both rpl 
semi sync master enabled and rpl semi sync slave enabled 





system variables disables semisynchronous replication where a value of 
one (1) will enable it. Semisynchronous replication is a dynamic process 
which you can toggle at runtime, for example. 

On the master host: 


master» SET GLOBAL rpl semi sync master enabled - 1; 


master» SHOW GLOBAL VARIABLES LIKE 'rpl semi sync master enabled'NG 
kckckckckckckckckckckckckckck ck ck ck ck ck ck ck k k k kk |l. row BERR KKK KKK ckckckckckckckckckck ck kk k kk 








Variable name: rpl semi sync master enabled 
Value: ON 





On the slave host: 


Slave» SET GLOBAL rpl semi sync slave enabled - 1; 


Slave» SHOW GLOBAL VARIABLES LIKE 'rpl semi sync slave enabled'\G 
Ckckckckckckckckckckckckckckckckckckckckc kc kc ck k k kk 7. row kkckckckckckckckckckckckckck ck ck ck kc kc kc kc k kk 








Variable name: rpl semi sync slave enabled 
Value: ON 





When enabled, this will be reported as a note in the MySOL error log: 


120617 18:51:08 [Note] Semi-sync replication initialized for transactions. 
120617 18:51:08 [Note] Semi-sync replication enabled on the master. 
Another system variable you will need to consider is rpl semi sync 
master timeout. This variable represents the time in milliseconds the 
master will wait on a commit acknowledgement from a slave host before 
timing out and switching to asynchronous replication. The default value is 
10,000, or 10 seconds, which is too high for many installations. In the 
example below the value has been changed from 10 seconds to 1 second: 


master» SET GLOBAL rpl semi sync master timeout - 1000; 


To ensure these dynamic semisynchronous replication settings are en- 
abled after a server restart, you will need to add the following to your 
configuration files. 

In the master configuration file: 

[mysqld] 


rpl semi sync master enabled 
rpl semi sync master timeout - 1000 


Il 
an 
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In the slave configuration file: 


[mysqld] 
rpl semi sync slave enabled = 1 





Semisynchronous replication currently includes four additional system 
variables. The full list is: 


master» SHOW GLOBAL VARIABLES LIKE 'rpl semi$'; 











Variable name Value 
rpl semi sync master enabled ON 

rpl semi sync master timeout 1000 
rpl semi sync master trace level 32 
rpl semi sync master wait no slave ON 


Monitoring the operations of semisynchronous replication is possible 
with a number of MySQL status variables: 


master» SHOW GLOBAL STATUS LIKE 'rpl$'; 


Rpl semi sync master clients 

Rpl semi sync master net avg wait time 

Rpl semi sync master net wait time 

Rpl semi sync master net waits 

Rpl semi sync master no times 

Rpl semi sync master no tx 

Rpl semi sync master status O 

Rpl_semi_sync_master_timefunc_failures 0 

Rpl semi sync master tx avg wait time 0 

Rpl semi sync master tx wait time 0 
0 
0 
0 
0 








Rpl semi sync master tx waits 

Rpl semi sync master wait pos backtraverse 
Rpl semi sync master wait sessions 

Rpl semi sync master yes tx 
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Semisynchronous replication can be disabled at any time when a time- 
out occurs. The Rpl semi sync master status status variable is im- 
portant to monitor: 


master» SHOW GLOBAL STATUS LIKE 'rpl$status'; 











In the event this occurs, the MySQL error log will report the situation, 
but not as an error: 


120617 18:53:09 [Warning] Timeout waiting for reply of binlog (file: 
alpha-bin.000004, pos: 357), semi-sync up to file , position O0. 
120617 18:53:09 [Note] Semi-sync replication switched OFF. 


Semisynchronous Operation 

Understanding the flow of a semisynchronous transaction will help in un- 
derstanding some of the drawbacks when using this feature. The primary 
steps are as follows: 


1. A slave connects to the master host. 
2. The slave declares to the master if it is using semisynchronous 
replication or not (in this case, we are). 


3. If both the master and at least one slave have semisynchronous 
replication enabled, transactions from the master to the slave will 
start using semisynchronous replication. 


4. A thread on the master runs a commit on a transaction. 


5. A wait occurs on the master while at least one semisynchronous 
slave acknowledges the transaction, or a timeout occurs if the wait 
on the master exceeds rpl semi sync master timeout. 


NOTE Ifa timeout occurs the master automatically reverts to asynchronous 
replication for future transactions. The master will automatically switch back 
to semisynchronous replication when the slave is caught up to the master host. 


6. On the slave, the transaction event is written to the relay log and 
flushed to disk. 
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7. The slave acknowledges to the master that the transaction is 
recorded. 


8. The master releases the thread that performed the transaction. 
9. The thread can execute the next transaction. 


As you may have seen in the process there are a few drawbacks when 
using semisynchronous replication. In the case where a system has more 
than one slave server, there is no confirmation all of the slaves have ac- 
knowledged the receipt of the event; only one slave has to acknowledge the 
event. On top of not waiting for all slave hosts to acknowledge receipt of 
events, semisynchronous replication does not require those events to be 
fully executed and committed on the slave host. Thus, it is still possible to 
see replication lag (Seconds Behind Master > 0) on the slave host 
and have the rpl semi sync master status variable set to ON in the 
master host. 

There are performance implications, as a commit has the overhead of 
slave acknowledgement for transactions. Network latency is an important 
factor when determining if semisynchronous replication is right for your 
installation and transaction throughput needs. It is still possible for replica- 
tion to revert back to asynchronous replication in the event of a slave error 
or slow network connection. Semisynchronous replication is a positive step 
toward better data integrity, and like other technologies there are tradeoffs 
that need to be considered specifically for your installation. In a more com- 
plex MySQL topology it is possible to have both semisynchronous and 
asynchronous replication between different MySOL instances. For exam- 
ple, in a multi-master topology, this may be semisynchronous replication, 
while additional attached slaves could use asynchronous replication. 


Synchronous Replication 


The standard MySQL server distribution does not currently support syn- 
chronous replication. MySOL does provide a different product called 
MySQL Cluster that does support this functionality. It is important to un- 
derstand that MySQL Cluster is a different product than MySQL server. 
An overview of MySOL Cluster is provided in Chapter 6. 

Third party products, including Galera, Tungsten, and Schooner Tech, 
are now offering MySOL synchronous replication features. The cloud 
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offerings of Amazon RDS with a Multi-AZ deployment and Google Cloud 
SQL also provide proprietary synchronous replication features. These will 
be discussed in detail in Chapter 6. 


Securing Replication with SSL 


Securing MySQL communication and MySQL replication with SSL has been 
around for a very long time; however, this is generally not implemented. 
With more organizations implementing cloud services, this additional secu- 
rity consideration is very significant for using replication. Setting up MySOL 
replication using an SSL connection for encryption will make it difficult for 
third parties to sniff out data transferred between the master and slave. 


Making MySOL SSL Ready 


You need to ensure SSL is enabled on both the master and slave host. To 
see if SSL is enabled, run the following command on both hosts: 


have openssl YES 
have ssl YES 
ssl ca 

Ssl capath 
Ssl cert 
Ssl cipher 
ssl key 











have openssl YES 
have ssl YES 
ssl ca 

Ssl capath 

Ssl cert 


Ssl cipher 
ssl key 
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Ifeitherhave opensslorhave sslissetto DISABLED, you will need 
to add the ss1 option to the my . cnf file and restart the MySQL server. 


Creating the Necessary Security Certificates 
The next step is to create and store three types of certificates, the Certifi- 
cate Authority (CA), server, and client. In this example we will create the 
certificates on the master host first in the /usr/local/mysql/certs 
directory that we will also create. 

On the master host: 


$ sudo mkdir -p /usr/local/mysql/certs 
$ openssl genrsa 2048 » ca-key.pem 
Generating RSA private key, 2048 bit long modulus 


eot 
e is 65537 (0x10001) 


$ openssl req -new -x509 -nodes -days 9999 \ 
-key ca-key.pem » ca-cert.pem 
You are about to be asked to enter information that will be incorporated 
into your certificate request. 
What you are about to enter is what is called a Distinguished Name or a DN. 
There are quite a few fields but you can leave some blank 
For some fields there will be a default value, 
If you enter '.', the field will be left blank. 
Country Name (2 letter code) [GB]:US 
State or Province Name (full name) [Berkshire]:CA 
Locality Name (eg, city) [Newbury]:Sunnyvale 
Organization Name (eg, company) [My Company Ltd] :EffectiveMySQL 
Organizational Unit Name (eg, section) []:SSL Replication 
Common Name (eg, your name or your server's hostname) []: 
Email Address []: 
You can see the two files created for the CA certificate below: 
$ ls *.pem 
ca-cert.pem ca-key.pem 


Next we need to create the server certificates: 


$ openssl req -newkey rsa:2048 -days 1000 -nodes -keyout \ 
server-key.pem > server-req.pem 
Generating a 2048 bit RSA private key 


You are about to be asked to enter information that will be incorporated 
into your certificate request. 
What you are about to enter is what is called a Distinguished Name or a DN. 
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There are quite a few fields but you can leave some blank 

For some fields there will be a default value, 

If you enter '.', the field will be left blank. 

Country Name (2 letter code) [GB] :US 

State or Province Name (full name) [Berkshire]:CA 

Locality Name (eg, city) [Newbury]:Sunnyvale 

Organization Name (eg, company) [My Company Ltd]:EffectiveMySOL 
Organizational Unit Name (eg, section) []:SSL Replication 
Common Name (eg, your name or your server's hostname) []: 

Email Address []: 


Please enter the following 'extra' attributes 
to be sent with your certificate request 

A challenge password []: 

An optional company name []: 


Leave the challenge password empty in the previous example. 


$ openssl x509 -req -in server-req.pem -days 9999 -CA ca-cert.pem \ 
-CAkey ca-key.pem -set serial 01 » server-cert.pem 

Signature ok 

subject=/C=US/ST=CA/L=Sunnyvale/O=EffectiveMySQL/OU=SSL Replication 

Getting CA Private Key 


You can see on the file system that three new files have been created: 


$ ls server-*.pem 
Server-cert.pem  server-key.pem  server-req.pem 


The last set of certificates is the client certificates, which are created as 
follows: 


$ openssl req -newkey rsa:2048 -days 9999 -nodes -keyout \ 
client-key.pem » client-req.pem 
Generating a 2048 bit RSA private key 


writing new private key to 'client-key.pem' 

You are about to be asked to enter information that will be incorporated 
into your certificate request. 

What you are about to enter is what is called a Distinguished Name or a DN. 
There are quite a few fields but you can leave some blank 

For some fields there will be a default value, 

If you enter '.', the field will be left blank. 

Country Name (2 letter code) [GB] :US 

State or Province Name (full name) [Berkshire]:CA 

Locality Name (eg, city) [Newbury]:Sunnyvale 

Organization Name (eg, company) [My Company Ltd]:EffectiveMySOL 
Organizational Unit Name (eg, section) []:SSL Replication 

Common Name (eg, your name or your server's hostname) []: 

Email Address []: 
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Please enter the following 'extra' attributes 
to be sent with your certificate request 

A challenge password []: 

An optional company name []: 


Leave the challenge password empty in the preceding example. 


$ openssl x509 -req -in client-req.pem -days 9999 -CA ca-cert.pem -CAkey \ 
ca-key.pem -set serial 01 > client-cert.pem 

Signature ok 

subject=/C=US/ST=CA/L=Sunnyvale/O=EffectiveMySQL/OU=SSL Replication 

Getting CA Private Key 


All of the certificates we just created are listed here: 


$ sudo mv *.pem /usr/local/mysql/certs/ 
$ ls -1 /usr/local/mysql/certs 


total 32 

-rw-rw-r-- 1 uid gid 1229 Jun 11 16:23 ca-cert.pem 
-rw-rw-r-- 1 uid gid 1679 Jun 11 16:23 ca-key.pem 
-rw-rw-r-- 1 uid gid 1099 Jun 11 16:25 client-cert.pem 
-rw-rw-r-- 1 uid gid 1704 Jun 11 16:24 client-key.pem 
-rw-rw-r-- 1 uid gid 956 Jun 11 16:24 client-req.pem 
-rw-rw-r-- 1 uid gid 1099 Jun 11 16:24 server-cert.pem 
-rw-rw-r-- 1 uid gid 1700 Jun 11 16:21 server-key.pem 
-rw-rw-r-- 1 uid gid 956 Jun 11 16:23 server-req.pem 





You will need to create the /usr/local/mysql/certs directory on 
the slave host and copy the newly created ca-cert.pem client-cert 
.pem client-key.penm files to the slave. 

Slave: 


$ sudo mkdir -p /usr/local/mysql/certs 
Master: 


$ scp ca-cert.pem client-cert.pem client-key.pem \ 
rootGslave.example.com:/usr/local/mysql/certs 
In the event the slave host is promoted to master, you will need to have 
all of the server SSL keys on the slave host. You can copy all files to the 
slave server. The previous scp syntax is to illustrate the minimum number 
of files needed. 


MySQL SSL Configuration Requirements 
The final step is to define these certifications in the master and slave 
my.cnf configuration file: 
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NOTE It is important to add the following configuration settings to both the 
master and the slave for failover purposes. 


Add the following variables to the [mysqld] section of the my. cnf file: 


$ vi /etc/my.cnf 

[mysqld] 

ssl 

ssl-ca=/usr/local/mysql/certs/ca-cert.pem 
ssl-cert=/usr/local/mysql/certs/server-cert.pem 
ssl-key=/usr/local/mysql/certs/server-key.pem 


A MySQL restart is needed on both the master and slave host for the 
SSL changes to take effect. The following example verifies the new con- 
figuration is operational: 


$ service mysql restart 
$ mysql -uroot -p -e "SHOW VARIABLES LIKE '$ssl$'" 











+--------------- +---------------------------------------- + 
Variable_name Value 
+--------------- eee + 
have openssl YES 
have ssl YES 
SSsl ca /usr/local/mysql/certs/ca-cert.pem 
ssl_capath 
ssl_cert /usr/local/mysql/certs/server-cert.pem 
ssl_cipher 
ssl_key /usr/local/mysql/certs/server-key.pem 
+--------------- +---------------------------------------- + 


MySQL User Privileges Requirements 

Replication requires a MySQL user account with the appropriate 
REPLICATION SLAVE permission. To use SSL, the REQUIRE SSL attri- 
bute is also necessary. 


NOTE Setting up a replication user with the requirement to use SSL is 
important. If you include REQUIRE SSL in your GRANT statement, only 
encrypted SSL slave threads will be allowed. In the following example, this 
means the user “repl” from % (everywhere) will only be able to connect to the 
master host with SSL encryption. If REQUIRE SSL is omitted, then both SSL 
encrypted and normal replication threads will be allowed with the same user. 
Here is a GRANT statement that illustrates how to enable a replication user, 
with SSL required for connections: 
master» GRANT REPLICATION SLAVE ON *.* TO 'repl@'%' 

-> IDENTIFIED BY 'some password' REQUIRE SSL; 
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It is also important to mention that if you have more than one GRANT 
set up for replication, you will need to modify these users to also use 
REQUIRE SSL. This would be the case if you have specifically added mul- 
tiple^repl"accounts from a specific IE IP range, or hostname. For example, 
if you have three slave hosts that connect to this master and have specified 
the slave's IP address to connect from, all specified"repl"users would need 
to have REQUIRE SSL added: 


master» SELECT user,host,password 
-» FROM mysql.user 
-» WHERE user - 'repl' 
-> ORDER BY host\G 


ckckckckck ck ck ck ckckckckckckck ck k k kk kkk |. row Kk k kk ckckckckckckckckck ck kc kk kk eA 


user: repl 
host: $ 
password: *48924CC1D59E9904D72265EFD60FB3C5C88BBEB5 


ckckckckck ck ck ck ckckckckckckckck ck k k kk kk DO. row KER KKK ck ck ck ck ckckckckckckckck kk kk kA 


user: repl 
host: 192.168.1.101 
password: *48924CC1D59E9904D72265EFD60FB3C5C88BBEB5 


ckckckckck ck ck ckckckckckckckckck ck k k kk kk 2. row kk KKK ck ck ck ck ck kckckckckckck ck ck kk kA 


user: repl 
host: 192.168.1.102 
password: *48924CC1D59E9904D72265EFD60FB3C5C88BBEB5 


ckckckckck ck ck ckckckckckckckckck ck k k kk kk A4. row kk ck ck ck ck ck kckckckck ck ck k kk kk eA 
user: repl 
host: 192.168.1.103 

password: *48924CC1D59E9904D72265EFD60FB3C5C88BBEB5 

4 rows in set (0.00 sec) 

Limiting replication connectivity to a specific IP is something you may 
want to implement in your environment when using SSL encrypted repli- 
cation. When implementing this you should also consider removing"repl" 
users with broader access credentials. Doing this can add a more granular 
level of security to your system. 

Adding the REQUIRE SSL option to any user can be accomplished by 
running the GRANT line shown earlier with the username and host modi- 
fied to fit our needs. In the following example the new privilege is defined 
for one user. The process would need to be repeated for all^repl"users that 
require SSL encryption. 
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Before: 


master» SHOW GRANTS FOR repl@192.168.1.102\G 


ckckckckck ck ck ckckckckckckckck ck k k kkkkk |. row Kk kk ck ck ck ck kckckckckck ck ck kk kk eA 


GRANT REPLICATION SLAVE ON *.* TO repl@192.168.1.102 
IDENTIFIED BY PASSWORD 
'"*48924CC1D59E9904D72265EFD60FB3C5C88BBEBS ' 


Adding the REQUIRE SSL option: 


master> GRANT REPLICATION SLAVE ON *.* TO 
reple192.168.1.102 
-» IDENTIFIED BY 'some password' REQUIRE SSL; 


After: 


master» SHOW GRANTS FOR rep1l@192.168.1.102\G 


KR KKK KK AK ck ckckckckckockckck ck ck kk kk TL LOW Ck ck ck ck ckckckck KKK ck ck ckck ck kok ck ck ck k k 
GRANT REPLICATION SLAVE ON *.* TO 

rep1@192.168.1.102 IDENTIFIED BY PASSWORD 
'*48924CC1D59E9904D72265EFD60FB3C5C88BBEB5' REQUIRE SSL 


On the slave host you have two options to start using SSL replication. 
You can either add the slave certificates to the [client] section in the 
slave my . cnf file, or you can explicitly specify the SSL information using 
the CHANGE MASTER TO statement. Remember, we copied the ca-cert 
.pem, client-cert.pem, and client-key.pem over to the slave host and these 
are located in the /usr/local/mysql/certs directory on the slave. The 
following are examples of both methods: 


NOTE Ifyou choose to implement the SSL changes in the [client] section of 
the my. cnf file you should also add these options to the master my . cnf file 
in the event the master is demoted to slave. 


Master and slave my. cnf file change: 


[client] 

ssl 

ssl-ca=/usr/local/mysql/certs/ca-cert.pem 
ssl-cert=/usr/local/mysql/certs/client-cert.pem 
ssl-key=/usr/local/mysql/certs/client-key.pem 


You then need to restart the slave host with --skip-slave-start and 
then run a CHANGE MASTER TO statement with the MASTER_SSL = 1 
option: 
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Slave» CHANGE MASTER TO 


MASTER HOST-'master.example.com'', 


MASTER USER-'repl', 


MASTER PASSWORD-'some password', 


MASTER PORT-3306, 


MASTER LOG FILE-'binary-logs.000001', 


MASTER LOG POS-106, 





MASTER CONNECT RETRY=10, 


MASTER SSL - 1; 


The second option is to specify all of the certificates within the CHANGE 


MASTER TO statement: 


Slave» CHANGE MASTER TO 





MASTER HOST-'master.example.com', 


MASTER USER-'repl', 


MASTER PASSWORD-'some password', 


MASTER PORT-3306, 








MASTER LOG FILE='binary-logs.000001', 





MASTER LOG POS-106, 
MASTER CONNECT RETRY-10, 
MASTER SSL - 1, 





MASTER SSL CA - '/usr/local/mysql/certs/ca-cert.pem', 
MASTER SSL CERT - '/usr/local/mysql/certs/client-cert.pem', 
MASTER SSL KEY - '/usr/local/mysql/certs/client-key.pem'; 


After you have your new SSL options in place, remember to run START 
SLAVE on the slave host and check that replication is running as it should be: 


Slave» START SLAVE; 

Slave» SHOW SLAVE STATUS\G 
Ckckckckckckckckckckckckckck ck ck ck ck ck k k kk kk*k*k 1. 
Slave IO State: 
Master Host: 
Master User: 
Master Port: 
Connect Retry: 
Master Log File: 
Read Master Log Pos: 
Relay Log File: 
Relay Log Pos: 
Relay Master Log File: 
Slave IO Running: 
Slave SQL Running: 








Master SSL Allowed: 
Master SSL CA File: 
Master SSL CA Path: 
Master SSL Cert: 
Master SSL Cipher: 
Master SSL Key: 


LOW **kkkkckckckckckckckckck ck ckckckckckck ck k k kk 
Waiting for master to send event 
master.example.com 

repl 

3306 

10 

binary-logs.000001 

106 

mysqld-relay-bin.000001 

4 

binary-logs.000001 

Yes 

Yes 


Yes 
/usr/local/mysql/certs/ca-cert.pem 


/usr/local/mysql/certs/client-cert.pem 


/usr/local/mysql/certs/client-key.pem 
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SSL replication is a long process and is difficult to diagnose if you mistype 
something or miss a step. Anyone who is looking for added security, espe- 
cially within a cloud service like Amazon Web Services (AWS), should con- 
sider running SSL replication for MySQL. More information can be found 
at http://dev.mysql.com/doc/refman/5.5/en/replication-solutions-ssl.html. 


New Replication Features 


Starting with MySOL 5.5, there have been major performance and scalabil- 
ity improvements added to the MySQL server. With MySQL 5.6 there are 
many new additions to replication specifically focused on data integrity, 
performance, and usability. 


New and Improved Data Integrity 


Data integrity is very important to a business and should not be taken 
lightly by database administrators. When data integrity issues are identi- 
fied, it usually takes operational time to remedy and engineering time to 
fix moving forward. With MySQL 5.6 less administration time is needed if 
a slave crashes, a human or application error occurs with a destructive SOL 
statement on a master, or you are suffering from data drift. 


Crash-Safe Slaves 

Traditionally, runtime replication information has been stored on the slave 
host in two files found in the data directory, master.info and relay- 
log.info. While this is still the default method in MySOL 5.6, MySOL 
now supports logging replication information to tables located in the 
mysql database. There are two new configuration variables that control 
how the master connection information and slave relay log information are 
stored: 


e master-info-repository When the value of this option is set 
to TABLE the master info log information will be stored in the 
mysql.slave master info table. When the value of this option is 
set to FILE the default filename master. info will be created on the 
file system to store the appropriate connection information. 
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€ relay-log-info-repository When the value of this option is 


set to TABLE the relay log information will be stored in the mysql 


.Slave relay log info table. When the value of this option is set 


to FILE the default filename relay-1log.info will be created on the 


file system to store the appropriate relay log information. 


NOTE It is important to review the grant privileges for all users. With these 


new tables, access to SELECT from the mysq1 schema would enable viewing 


the replication user password, which is stored in plain text. 


By default, these new tables exist in the mysql schema for MySQL 5.6; 
however, they contain no information: 


slave> SELECT * 
Empty set (0.00 
slave> SELECT * 
Empty set (0.00 


sec) 








sec) 


FROM mysql.slave master info; 


FROM mysql.slave relay log info; 


The following configuration is enabled on a slave to demonstrate the 


information available: 


[mysqid] 


master-info-repository-TABLE 
relay-log-info-repository-TABLE 


Slave» SELECT * FROM mysql.slave master infoNG 


kckckckckckckckckckckckckckck ck ck ck ck ck k k kk kkk 1. 


Master id: 
Number of lines: 
Master log name: 

Master log pos: 
Host: 

User name: 

User password: 
Port: 

Connect retry: 
Enabled ssl: 

Ssl ca: 

Ssl capath: 

Ssl cert: 

Ssl cipher: 

Ssl key: 

Ssl verify server cert: 
Heartbeat: 

Bind: 

Ignored server ids: 
Uuid: 

Retry count: 

Ssl crl: 

Ssl crlpath: 
Enabled auto position: 








LOW kk kk ck ck ck kckckckckckckckckckckckckck ck ck KEE 
3 

23 

alpha-bin.000003 

187 

alpha 

repl 

*clear password text* 
3306 

60 

0 


1800 


ba7ac732-b707-11e1-a1b3-0800275824dc 
86400 
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mysql» SELECT * FROM mysqgl.slave relay log info\G 
kckckckckckckckckckckckckckck ck ck ck ck ck kc ck ck k k kk |l. row '*kkckckckckckckckckckckckckck ck ckckckck kc kk kk 
Master id: 3 
Number of lines: 6 
Relay log name: ./gamma-relay-bin.000006 
Relay log pos: 346 
Master log name: alpha-bin.000002 
Master log pos: 391 
Sql delay: 0 
Number of workers: 0 
1 row in set (0.00 sec) 


The default storage engine for the two new tables defined with slave __ 
master infoand slave relay log info is MyISAM. It is important 
to note that you will need to change the engine of these tables to InnoDB in 
order for replication to be crash safe. It is recommended you stop MySOL 
replication during this process. For example: 


Slave» STOP SLAVE; 

Slave» ALTER TABLE mysql.slave master info ENGINE = InnoDB; 
Slave» ALTER TABLE mysql.slave_ relay log info ENGINE = InnoDB; 
Slave» START SLAVE; 























NOTE MySQL will not operate if you attempt to change the storage engine 
from MyISAM for any other mysq1 schema tables. 


Keep in mind that you can use other storage engines for these tables; 
however, the same transactional engine should be used so replication in- 
formation can be added to transactions and committed with the corre- 
sponding transaction. More information can be found at http://dev.mysql 
.com/doc/refman/5.6/en/slave-logs.html. 


Delayed Replication 

Time delayed replication has been used through the history of MySQL 
replication for disaster recovery purposes and testing application behavior 
while a slave is lagging. Making a slave host in a MySQL topology lag be- 
hind by a certain amount of time can help avoid catastrophic operational 
errors introduced on the master, for example, a TRUNCATE TABLE. In the 
event a TRUNCATE TABLE command was issued on the master host, an 
administrator would be able to skip the physical statement and then pro- 
mote the delayed slave to master. This new feature is implemented with 
the CHANGE MASTER TO statement and the MASTER_DELAY attribute: 
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e MASTER DELAY This attribute specifies how much time in seconds 
the SOL THREAD will pause execution of events on the slave host. 
The default value is zero (0), or no lag, with an upper bound of 68 
years, or 23!— 1. 


A time delay can be set on any slave server and acts independently of 
other replication streams. An administrator could set a time delay during 
the initial CHANGE MASTER TO statement or after replication has started 
with the following commands: 


Slave» STOP SLAVE; 
Slave» CHANGE MASTER TO MASTER DELAY - 10; 
slave» START SLAVE; 





You can observe time delay in action with the SHOW SLAVE STATUSNG 
command on the slave. For example: 


Slave» SHOW SLAVE STATUS\G 
ckckckckckckckckckckckckckck kc k k k kk kk |l. row KEKE KK k kk ck kk kk KKK KKK KKK 
Slave IO State: Waiting for master to send event 
Master Host: master.example.com 


SQL Delay: 10 
SQL Remaining Delay: 6 
Slave SQL Running State: Waiting until MASTER DELAY seconds after 
master executed event 


The SQL Delay column shows the desired lag time in seconds; in this 
example, we want ten seconds of delay. The SOL, Remaining Delay indi- 
cates how many seconds the slave needs to wait until new events are ap- 
plied to the slave. The S1ave SQL Running State corresponds to the 
Slave IO State and displays the same information shown with SHOW 
FULL PROCESSLIST: 


Slave» SHOW FULL PROCESSLIST\G 
okckckckckckckckck ck ck k ck k kk kk kkkk 1, row XX kk kk k kk k KK kk KK KEKE 
Id: 10 
User: system user 
Host: 
db: NULL 
Command: Connect 
Time: 1414 
State: Waiting for master to send event 
Info: NULL 
kckckckckckckckckckckckckckck ck ck ck k k k kk 2. row kk kckckckckckckckckckckckck kckck ck KK 
Id: 11 
User: system user 
Host: 
db: NULL 
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Command: Connect 
Time: 10 
State: Waiting until MASTER DELAY seconds after master executed event 
Info: NULL 

okckckckckckckckckckckckckck kck ck kc k k kk 3. row FRR KKK KKK KKK KKK KKK KK EK KE 


Id: 12 
User: root 
Host: localhost 
db: NULL 
Command: Query 
Time: 0 
State: init 
Info: SHOW FULL PROCESSLIST 
More information can be found at http://dev.mysql.com/doc/refman/5.6/ 


en/replication-delayed.html. 


Checksums for Replicated Data 
Prior to version 5.6 an administrator would have to use SSL replication as 
a workaround to implement checksums. With the implementation of 
CRC32 checksums, you can be assured that data being replicated to a slave 
has the same integrity as that applied to the master. If data corruption 
occurs an error will be returned and slave replication is stopped. Checksum 
information is recorded in the master binary log and the slave relay logs 
and can be activated on a per server basis. 

To enable checksums with the master binary log, the following setting is 
needed: 


€ binlog checksum When this option is set to CRC32 the master 
will write a checksum for each event into the binary log. binlog_ 
checksum is a dynamic global variable with the default value of 
NONE. Changing this option to CRC32 will rotate the binary log, as 
checksums are always written to an entire binary log file. If the default 
is set for this option, the server will verify that only complete events 
are written to the binary log by writing and checking the event length. 


Enabling binlog checksum does not force verification of the check- 
sums created. There are two system variables, one for the master host and 
the other for slave host(s) to enforce the verification of CRC32 checksums. 

On the master host: 


e master verify checksum This variable is disabled by default. 
When this variable is set to one (1) or ON, the master host will 
examine checksums when reading from the binary log. 
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On the slave host: 


€ slave sql verify checksum This option is also disabled by 
default. When slave verify checksum is enabled with a value of one 
(1) or ON, the slave host will examine and verify checksums when 
reading the relay log. 


When enabled, the binary log provides the following information: 


$ mysqlbinlog binary-logs.000008 

#111129 22:29:25 server id 1 end_log_pos 584974 CRC32 0x3e864aba 
Query thread_id=19 exec_time=0 error code-0 

SET TIMESTAMP=1322605765/*!*/; 

INSERT INTO 

/*1*f; 

# at 584974 

#111129 22:29:25 server id 1 end log pos 585005 CRC32 0x1376938a 
Xid - 24943 

COMMIT/*!*/; 

DELIMITER ; 

# End of log file 

ROLLBACK /* added by mysqlbinlog */; 

/*150003 SET COMPLETION TYPE-GOLD COMPLETION TYPE*/; 








The following details are recorded in the slave relay log for comparison: 


$ mysqlbinlog mysqld-relay-bin.000007 

#111129 22:29:25 server id 1 end_log pos 584974 CRC32 0x3e864aba 
Query thread_id=19 exec_time=0 error code-0 

SET TIMESTAMP=1322605765/*!*/; 

INSERT INTO 

# at 585133 

#111129 22:29:25 server id 1 end log pos 585005 CRC32 0x1376938a 
Xid - 24943 

COMMIT/*!*/; 

DELIMITER ; 

# End of log file 

ROLLBACK /* added by mysqlbinlog */; 

/*150003 SET COMPLETION TYPE-GOLD COMPLETION TYPE*/; 

















More information can be found at http://dev.mysql.com/doc/refman/5.6/ 
en/replication-options-binary-log.html#option_mysqld_binlog-checksum. 


New Performance Improvements for Replication 


Two new performance related additions in MySQL 5.6 for replication are 
multi-threaded slaves and row image control within row-based replication 
(RBR). 
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Multi-Threaded Slaves 
Parallel transaction execution on a slave host provides better scalability in 
replication, especially on multicore systems, but there is a catch. 

The SOL thread acts as a coordinator for what are referred to as slave 
worker threads, which work on a per database schema level and can be 
enabled and tuned with the slave parallel workers system variable. 
To utilize multi-threaded slaves, your data will need to be partitioned into 
multiple database schemas. This allows worker threads on the slave to pro- 
cess transactions on a given database without blocking when updates are 
processed on others. You also need to ensure there are no foreign key de- 
pendencies across schemas and there are no cross schema DML queries 
(i.e,, INSERT INTO db1.t1 SELECT * FROM db2.t2). 

When slave parallel workers is set to a non-zero value on the 
slave, host transactions will not necessarily be applied on the slave in the 
same order as they were recorded in the master host binary log. This can 
lead to recovery complexity along with interpreting relay log information. 
This variable is defined in the my.cnf configuration file. A MySOL 
instance restart is necessary for these configuration settings to take effect: 

[mysqld] 
Slave parallel workers-3 
relay log info repository-TABLE 

With the implementation of slave parallel workers a new table 
has been added to the mysq1 database, s1ave worker info. Along with 
the new crash-safe tables you should also convert the slave worker 
info table to InnoDB: 


master» ALTER TABLE mysql.slave worker info ENGINE-InnoDB; 


Parallel threads will only work when both master and slave hosts are 
MySQL version 5.6 or higher. If you attach a version 5.6 slave to a version 
5.5 master, replication will slow down exponentially when the multi- 
threaded configuration is enabled on the slave. 

When running a test on multiple schemas using the example procedure 
to test replication found in the appendix, you can observe the following 
information: 


Slave» SELECT * FROM mysql.slave worker infoNG 
kckckckckckckckckckckckckckck ck ckck kc k kc k k kk kk 1. row kk kckckckckckckckckckckckckckckckck ck ko kk 
Master id: 3 
Worker id: 0 
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Relay log name: 
Relay log pos: 
Master log name: 
Master log pos: 
Checkpoint relay log name: 
Checkpoint relay log pos: 
Checkpoint master log name: 
Checkpoint master log pos: 
Checkpoint seqno: 
Checkpoint group size: 
Checkpoint group bitmap: 
kckckckckckckckckckckckckck ck ckckckck kckc kc k kk kk 
Master id: 

Worker id: 

Relay log name: 
Relay log pos: 
Master log name: 
Master log pos: 
Checkpoint relay log name: 
Checkpoint relay log pos: 
Checkpoint master log name: 
Checkpoint master log pos: 
Checkpoint seqno: 
Checkpoint group size: 
Checkpoint group bitmap: 
kckckckckckckckckckckckckckckckckckckckckck kc k kk k 
Master id: 

Worker id: 

Relay log name: 
Relay log pos: 
Master log name: 
Master log pos: 
Checkpoint relay log name: 
Checkpoint relay log pos: 
Checkpoint master log name: 
Checkpoint master log pos: 
Checkpoint seqno: 
Checkpoint group size: 
Checkpoint group bitmap: 


3 rows in set (0.00 sec) 
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Q. LOW BERK RRR KEK RRR KARE RRR kkk kkk 


3. LOW FRR R kk k k KR k k k k k k k KR k k k k k k k 


2 
./gamma-relay-bin.000009 
13641 

alpha-bin.000003 

21209 
./gamma-relay-bin.000009 
13329 

alpha-bin.000003 

20897 

0 
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More information can be found at http://dev.mysql.com/doc/refman/5.6/ 


en/replication-options-slave.html£option mysqld slave-parallel-workers. 


Row-Based Replication - Row Image Control 

Row-based replication (RBR), when compared to statement-based replica- 
tion (SBR), has traditionally taken up more space in the binary log. This is 
due to the default behavior of RBR where all column values are sent to the 
slave where a write occurs instead of just sending the values in the row 
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where the write occurs. With row image control it is now possible to save 
system and network resources by toggling the behavior of row-based rep- 
lication. Detailed next are examples of all three settings for the new system 
variable ,binlog row image. 

First we need to create a table, in the following example, called test 
rbr image: 


master» use book3 

master» CREATE TABLE test rbr image ( 
-» id INT UNSIGNED NOT NULL, 
-> coll INT DEFAULT NULL, 
-> var2 CHAR(3) DEFAULT NULL, 
-» comment TEXT NULL 

) ENGINE-InnoDB DEFAULT CHARSET-utf8; 


When the binlog row image system variable is set to full (the de- 
fault), all columns will be logged to the binary log after a write event. With 
the INSERT statement, we are adding a row to the table but only adding 
values to three (3) columns to the row: 


master» SET SESSION binlog format - ROW; 
master» SHOW MASTER STATUS\G 
ckckckckck ck ck ckckckckckckckck ck k k kkkkk |. row Kk kk kk kk kckckckckck ck ck kk kk A 
File: mysql-bin.000002 
Position: 11783 
Binlog Do DB: 
Binlog Ignore DB: 
Executed Gtid Set: 
master» INSERT INTO test rbr image (id, coll, var2) 
-> VALUES(1, 2, '3'); 


Now we can see what this event looks like in the binary log using the 
mysqlbinlog client utility and the values from the output of SHOW 
MASTER STATUS: 


$ mysqlbinlog --base64-output-DECODE-ROWS \ 
--verbose --start-position-11783 mysql-bin.000002 
# at 11783 


# at 11912 

#120613 17:11:10 server id 1 end log pos 11912 

Table map: ^book3'^. "test rbr image mapped to number 73 
#120613 17:11:10 server id 1 end log pos 11952 
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Write rows: table id 73 flags: STMT END F 
### INSERT INTO book3.test rbr image 


### SET 

### 01-1 
dH @2=2 
### 03-2'3' 


Hit @4=NULL 


Although we only added three column values to the row, all four of the 
column values end up in the binary log. The value for the fourth column in 
the row (in bold), in this case column comment, is set to NULL even though 
we did not reference the column in the INSERT statement. 

When binlog_row_image is set to minimal, only the changed columns 
are added to the binary log. The following is an example SQL statement 
with this format: 


master» SET SESSION binlog row image-minimal; 
master» SHOW MASTER STATUS\G 
ckckckckck ck ck ckckckckckckckck ck k kk kkkk |. row kk kckck ck ck k ck ck ck ck ck ck kk k kk eA 
File: mysql-bin.000002 
Position: 12169 


master» INSERT INTO test rbr image(id) VALUES(4); 


Let's take a look inside the binary log using the mysqlbinlog client 
utility: 


$ mysqlbinlog --base64-output-DECODE-ROWS --verbose V 
--start-position=12169 mysql-bin.000002 


# at 12298 

#120613 17:22:43 server id 1 end_log_ pos 12298 

Table map: ^book3^.^test rbr image" mapped to number 73 
#120613 17:22:43 server id 1 end log pos 12332 

Write rows: table id 73 flags: STMT END F 

### INSERT INTO book3.test rbr image 


### SET 
THER @1=4 
# at 12332 


You can see there is only one (1) column specified in the binary log for 
this event. It is relatively easy to see that with this new behavior less disk 
space and network bandwidth will be used during replication. 
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The last setting for binlog row image is noblob.In this case all non- 
text or blob columns are omitted from the binary log unless those columns 
are being added or changed. For example: 


master» SET SESSION binlog row image-noblob; 
master» SHOW MASTER STATUS\G 


kckckckckckckckckckckckckckckckckckckck kc kc ck k k kk 7. row *kkckckckckckckckckckckckck ck ckckckckckc kc kc k kk 
File: mysql-bin.000002 
Position: 12359 


master» INSERT INTO test rbr image(id,var2) VALUES (5,'abc'); 
$ mysqlbinlog --base64-output-DECODE-ROWS --verbose \ 

--start-position-12359 mysql-bin.000002 
# at 12488 
#120613 17:44:48 server id 1 end log pos 12488 
Table map: ^book3^.^test rbr image" mapped to number 73 
#120613 17:44:48 server id 1 end log pos 12526 
Write rows: table id 73 flags: STMT END F 
### INSERT INTO book3.test rbr image 
### SET 
HHH @1=5 
### =  @2=NULL 
HHH @3='abc' 

It is also important to mention that when using minimal or noblob for 
binlog row image the tables from the master host to the slave(s) need to 
have the same columns in the same order with the same data type and 
identical primary keys. 

More information can be found at http://dev.mysql.com/doc/refman/5.6/ 
en/replication-options-binary-log.htmlitsysvar binlog row image. 


New Replication Management Features 


The MySQL server and client utilities have always been easy to use, and 
now there are two additions that make administration more robust. Binary 
log backups through the mysqlbinlog client utility and Universally 
Unique Identifiers (UUID) are now available in MySQL 5.6. 


Remote Binary Log Backup 

When running point in time recovery it is necessary to save the master bi- 
nary logs.This is generally implemented by copying binary logs to a shared 
storage device or another server. The mysqlbinlog client utility now has 
this functionality built in, making it easier to copy binary logs to different 
servers and locations. 
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There are several new options for mysqlbinlog that are used to back 


up binary logs to a second server: 


i 


e --raw This option ensures that the copied binary log will be in the 
same format as the original and not in text format. 


€ --read-from-remote-server | -R Thisoption tells 
mysqlbinlog to connect to a server and request the binary log(s) 
from the host. 


€ --read-from-remote-master-type This option reads the 
binary logs. The value of BINLOG-DUMP-NON-GTIDS is equivalent to 
--read-from-remote-server. The other valid value is BINLOG- 
DUMP GTIDS. 


€ --to-last-log Will gather all binary logs starting with the 
binary log specified to the last binary log on the server. 


e --stop-never Will run mysqlbinlog in a daemon mode and 
stay connected to the server after reaching the end of the last log. If 
--stop-never is specified, it is not necessary to add the - -to- 
last-1og option, as this is implied with - -stop-never. 


e --results-file When added in conjunction with - - raw the 
value of this option will be added as a prefix name to the output files. 


In the following examples we will be using the nine (9) binary logs spec- 
fied here: 


master» SHOW BINARY LOGS; 














Log name File size 
binary-logs.000001 60286 
binary-logs.000002 909656 
binary-logs.000003 124878902 
binary-logs.000004 52616276 
binary-logs.000005 158611 
binary-logs.000006 700233141 
binary-logs.000007 18865438 
binary-logs.000008 342192894 
binary-logs.000009 1364 
be bee ees See eee eee RE MUS + 
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To make a static backup of all the binary logs, you can use the following 
command: 
$ mysqlbinlog -usomeuser --host=master.example.com \ 
--port=3306 --raw --read-from-remote-server \ 
--to-last-log binary-logs.000001 

On the host where the mysqlbinlog command was executed we can 
now see the binary logs on the file system: 


$ 1s -1 |awk '{print $9 " | " $5}! 
binary-logs.000001 60286 
binary-logs.000002 909656 
binary-logs.000003 124878902 
binary-logs.000004 52616276 
binary-logs.000005 158611 
binary-logs.000006 700233141 
binary-logs.000007 18865438 
binary-logs.000008 342192894 
binary-logs.000009 1364 








To run mysqlbinlog in daemon mode and keep a live running backup 
of all binary logs, you can use the following command: 


$ mysqlbinlog -uroot --host=master.example.com --port-3306 \ 
--raw --read-from-remote-server \ 
--stop-never binary-logs.000001 

The binary backups will continue to be recorded while mysqlbinlog is 
running with the --stop-never option. After some write operations on 
the master host you can see that the byte value of binary-logs.000009 
has changed to 585352 from its previous value of 1364 on the backup 


server: 

$ Is -l1 |awk '{print $9 " | " $5}! 
binary-logs.000007 | 18865438 
binary-logs.000008 | 342192894 
binary-logs.000009 | 585352 


Binary logs will continue to be recorded up until mysqlbinlog is 
stopped or the master server is shut down. Appropriate system monitoring 
should be in place if this daemon operation is implemented to ensure this 
does not stop unexpectedly. 

For more information visit http://dev.mysql.com/doc/refman/5.6/en/ 
mysqlbinlog.html. 
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Universally Unique Identifier (UUID) 

The UUID is now added to every server running MySQL version 5.6 and 
above. This addition allows easier tracking and auto discovery for remote 
monitoring and inventory systems.The new system variable server uuid 
is accessible with a SELECT statement and, if replication is turned on, 
through the SHOW SLAVE STATUS command. For example: 


master» SELECT @@GLOBAL.server_uuid; 








This information is stored in the auto. cnf file found in the data directory: 


$ cat auto.cnf 
[auto] 
server-uuid= cae53eb6-1laa8-11e1-a608-00238b979631 


Viewing the master host UUID on the slave host: 


slave> SHOW SLAVE STATUS\G 
kkkkkkkkkkkkkkkkkkkkkkk 1], COW KEK ck ck ck kk kkk ck ck ck ckck ck ck kkk kk 


Slave IO State: Waiting for master to send event 
Master Host: master.example.com 
Master User: repl 
Master Port: 3306 


Master Server Id: 1 
Master UUID: cae53eb6-1aa8-11e1-a608-002380979631 


For more information see http://dev.mysql.com/doc/refman/5.6/en/ 
replication-options.htmlfsysvar server uuid. 


Global Transaction Identifier (GTID) 
GTIDs make it easier to keep track of replication between a master and 
slave and with other topologies, including cascading and circular replica- 
tion. Building a mechanism in house to keep track of replication, run 
failovers, and promote slaves is no longer necessary with the new GTID 
utilities that accompany this new feature. 

Global transaction identifiers are composed of the original master host 
UUID and a sequence that is automatically generated for every event 
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written to the binary log. This means that every transaction written to the 
binary log has a unique GTID assigned to it, making it very easy to track 
and compare the progress of replicated events on every slave host in a rep- 
lication topology. To enable GTID usage you need all four of the following 
configuration variables in the my . cnf file: 

[mysqid] 

log-bin 

log-slave-updates 

gtid-mode-ON 

disable-gtid-unsafe-statements 





There are several limitations to GTID usage. GTID will not work when: 


e Using nontransactional statements, i.e., MyISAM tables, including 
for example running the mysql secure installation command 


e CREATE TABLE ... SELECT 
e Temporary tables within transactions 


This means that you will not be able to use GTID unless you are using 
InnoDB specifically. Using GTID-based replication is different from using 
traditional replication. When gtid mode is active on all master and slave 
servers you will be able to use MASTER AUTO POSITION in the 
CHANGE MASTER TO statement. MASTER. AUTO POSITION is new in 
MySQL 5.6.5 and utilizes the GTID on the master to synchronize slave 
servers within a replication topology. If you do choose to use MASTER -. 
AUTO POSITION, you will no longer be able to specify MASTER LOG . 
FILE and MASTER LOG POS in the CHANGE MASTER TO statement. 
Here is an example of MASTER AUTO POSITION usage: 


Slave» CHANGE MASTER TO 
=> MASTER HOST-'master.example.com', 
-> MASTER USER-'repl', 
zb MASTER PASSWORD-'somepassword, 
-> MASTER PORT-3306, 
-> MASTER AUTO POSITION = 1; 
Query OK, 0 rows affected, 2 warnings (0.24 sec) 


There are two warnings that have been added in version 5.6 when run- 
ning a CHANGE MASTER TO statement. The following is the output of 


SHOW WARNINGS after the CHANGE MASTER TO statement has been 
issued: 
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Slave» SHOW WARNINGS\G 
KR RRR RRR ck ck ckok kk kk ke RRR 1, LOW CK Ck ck ck kck ck ck kck ckock kck kock kck ck ck kk 


Level: Note 
Code: 1756 
Message: Sending passwords in plain text without SSL/TLS is extremely 


insecure. 
ckckckckckckckckckckckckckck ck ck ck k k k kk k 2., row X'**kckckckckckckckckckckckckckck kcko kk kk 


Level: Note 
Code: 1757 
Message: Storing MySQL user name or password information in the master.info 
repository is not secure and is therefore not recommended. Please see the 
MySQL Manual for more about this issue and possible alternatives. 


NOTE The output specifies MySQL best practices for securing replication and 
storing passwords. 


We can now start GTID-based replication on the slave host and view the 
corresponding replication thread connection on the master host: 


Slave» START SLAVE; 


master» SHOW PROCESSLIST\G 
ckckckckckckckckck kck ck ck ck k kk kkkkkk |. row ck ck ckockckckockockckckckckckckckckckock kc kk kk 
Id: 3 
User: repl 
Host: master.example.com:40709 


db: NULL 
Command: Binlog Dump GTID 
Time: 65 


State: Master has sent all binlog to slave; waiting 
for binlog to be updated 
Info: NULL 


NOTE You can see the Command column value now indicates that we are using 
GTID for replication. 


In the event you would like to move back to non-GTID replication you 
will need to disable MASTER AUTO POSITION and then run the correct 
CHANGE MASTER TO statement. The following example running GTID- 
based replication will show the error condition and correct syntax to switch 
back to non-GTID replication: 


slave» STOP SLAVE; 
slave> SHOW SLAVE STATUS\G 
Slave_IO State: 
Master Host: master.example.com 
Master User: repl 
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Master Port: 
Connect Retry: 
Master Log File: 
Log Pos: 
Relay Log File: 
Relay Log Pos: 

Relay Master Log File: 
Running: 
Running: 
Replicate Do DB: 

Replicate Ignore DB: 
Replicate Do Table: 
Replicate Ignore Table: 
Replicate Wild Do Table: 
Replicate Wild Ignore Table: 
Last Errno: 
Last Error: 
Skip Counter: 

Exec Master Log Pos: 
Relay Log Space: 
Until Condition: 

Log File: 
Log Pos: 
Master SSL Allowed: 
Master SSL CA File: 
Master SSL CA Path: 
Master SSL Cert: 
Master SSL Cipher: 
Master SSL Key: 

Seconds Behind Master: 

Master SSL Verify Server Cert: 
Last IO Errno: 
Last IO Error: 
jast SQL Errno: 
ASE SQL Error: 
Replicate Ignore Server Ids: 
Master Server Id: 

Master UUID: 


Read Master 





Slave IO 
Slave SQL 











Until 
Until 
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3306 

10 

binary-logs.000010 

193. 
mysqld-relay-bin.000003 
365 

binary-logs.000010 

No 

No 


f52e965a-bi8d-11el-bdfe-00238b979631 


Master Info File: 

SQL Delay: 

SOL Remaining Delay: 
Slave SOL Running State: 
Master Retry Count: 
Master Bind: 

Last IO Error Timestamp: 
Last SOL Error Timestamp: 
Master SSL Crl: 
Master SSL Crlpath: 
Retrieved Gtid Set: 
Executed Gtid Set: 





/local/mysql/slave/master.info 
0 
NULL 


86400 


8A94F357-AAB4-11DF-86AB-C80AA9429562:1-9, 
F52E965A-B18D-11E1-BDFE-00238B979631:1-155 
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Slave» CHANGE MASTER TO 
-> MASTER HOS ='master.example.com', 
-> MASTER_USER='repl', 


-> MASTER PASSWORD-'somepassword', 


-> MASTER PORT-3306, 
-> MASTER LOG FILE='binary-logs.000010', 
-> MASTER LOG POS-191, 

-> MASTER CONNECT RETRY-10; 
ERROR 1775 (HY000): Parameters MASTER LOG FILE, MASTER LOG POS, 
RELAY LOG FILE and RELAY LOG POS cannot be set when MASTER AUTO POSITION 


is active. 

















As you can see there is an error when you just issue a CHANGE MAS- 
TER TO statement. We will now set MASTER AUTO POSITION to 0 and 
re-run the CHANGE MASTER TO statement: 


Slave» CHANGE MASTER TO MASTER AUTO POSITION - 0; 
Slave» CHANGE MASTER TO 

-> MASTER HOST-'master.example.com', 

=> MASTER USER-'repl', 

-> MASTER_PASSWORD='somepassword', 

=> MASTER PORT=3306, 

x MASTER LOG FILE='binary-logs.000010', 

=> MASTER_LOG_POS=191, 

-> MASTER_CONNECT_RETRY=10; 
slave> START SLAVE; 

















We are now replicating with non-GTID replication. 

More information on GTID configuration setting can be found at http:// 
dev.mysql.com/doc/refman/5.6/en/replication-options-gtids.html. 

There are two new utilities, which help monitor and, if needed, promote 
or fail over to another server in your replication environment. These utili- 
ties are discussed in Chapter 5. 


Binary Log Group Commit 

The most recent version of MySQL 5.6.6 (2012-08-07) includes a feature to 
improve performance of binary log writing. By grouping several writes to- 
gether in a high volume environment, throughput can be greatly improved. 
This feature is very important when ensuring maximum durability via the 
sync binlog-1 setting. No additional configuration is necessary as this is 
enabled by default. This feature introduces the configuration variables bin- 
log order commits,binlog max flush queue time, and innodb _ 
flush log at timeout to provide additional compatibility and fine- 
tuning option. 
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More information including a detailed technical description can be found 
at http://mysqlmusings.blogspot.in/2012/06/binary-log-group-commit-in- 
mysql-56.html. 

Initial work on improving group commit was first included with MariaDB, 
a fully compatible version of MySQL. This work by Monty Program AB was 
provided as part of the MySQL open source license for Oracle or any other 
organization to implement or modify. More information on this work is 
available at http://kristiannielsen.livejournal.com/12254.html. 


Balancing Read and Write Load 


With the introduction of a MySQL topology using replication, your applica- 
tion may require modification to support read operations and write opera- 
tions with different connection parameters. Fortunately, several ofthe MySOL 
connectors provide this functionality natively to minimize the requirements 
for application modifications. 

Connector/J for Java has enabled the splitting of read and write SOL 
statements for many years. Recent improvements in Connector/J 5.1.12 and 
5.1.13 have further improved these load balancing features to include 
failover support. More information is available at http://dev.mysql.com/ 
doc/refman/5.0/en/connector-j-usagenotes-j2ee-concepts-managing-load- 
balanced-connections.html and http://dev.mysql.com/doc/refman/5.0/ en/ 
connector-j-usagenotes-j2ee-concepts-load-balancing-failover.html. 

The PHP native driver (mysqlInd) also provides appropriate connection 
management for reads and writes. Refer to http://dev.mysql.com/doc/ref- 
man/5.5/en/apis-php-book.mysqlnd-ms.html and http://blog.ulf-wendel 
.de/2012/peclmysqInd ms-14-a-failover-standby-using-weightedpriori- 
tized-load-balancing/ for more information. This driver is the default in 
PHP 5.4 and higher. You can also find information in the PHP reference 
manual at http://php.net/manual/en/mysqlInd-ms.rwsplit.php. 

MySQL proxy was another product to consider for load balancing. Avail- 
able from http://dev.mysql.com/downloads/mysgql-proxy/, this product has 
stalled in development in recent years and future work is undetermined. 
An example for load balancing and splitting connections with MySOL 
proxy can be found at http://agiletesting.blogspot.com/2009/04/mysql-load- 
balancing-and-read-write.html. 
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Additional products can be used to manage load balancing read con- 
nections to a pool of slaves. The primary concern is the necessary manage- 
ment and monitoring of replication lag, and the proactive removal of slave 
servers to ensure data is consistent. 


Conclusion 


This chapter has covered many new features of MySQL replication in 5.5 
and 5.6. For any large MySQL environment using replication with a previous 
version, these are significant reasons to consider upgrading and benefiting 
from performance, data integrity, network optimizations, and improved 
failover capabilities when replication is an important component of your 
MySQL topology. 

Examples and links in this chapter are available for download from 
http://EffectiveMySQL.com/book/replication-techniques. 
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Using Multi-Master 
Replication 


MySQL replication can be used for many benefits, including supporting 
scalability and higher availability. By default, MySOL replication does not 
support an environment that manages failover situations, or writing concur- 
rently to multiple servers in a replication topology, without additional 
configuration and management. 
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In this chapter we will discuss: 
e Configuring MySQL replication to support failover 
e Demonstrating the manual steps for failover and failback 


e More advanced replication topologies 


MySQL Replication Failover Capabilities 


A common way to set up a highly available (HA) MySQL environment is 
to use a multi-master topology with two servers. Multi-master, also 
known as a MySQL pair, is configured with two servers, and both servers 
can act as a master and slave in replication terms. In this situation this is 
an active/passive master configuration that is a common MySQL deploy- 
ment approach. A true active/active master/master MySOL configuration 
is when both servers are supporting write activity at the same time. This 
type of setup is not typically recommended, given the number of data and 
scalability problems you may encounter. Writing to both servers does not 
actually increase throughput but only adds a level of redundancy. This con- 
figuration makes failure situations more complex in order to support the 
same amount of writes on one server when the volume is supported with 
two servers under normal conditions. 

A better practice is to set up a multi-master environment with two servers, 
where writes are supported on one server at a time and the other server is 
acting as a standby that can support read load. 


Active/Passive Multi-Master Replication 


Multi-master replication is when each server replicates to the other, so 
server A would replicate to server B and server B would replicate to server 
A. An active/passive or active/standby configuration is a safe approach to 
run a multi-master environment to support higher availability. This means 
that one server, the active server, will support all write activity along with 
reads and the other server, the passive server, only supports read activity if 
necessary. The passive or standby server in this case will be ready and avail- 
able if the active server should fail or a planned failover occurs. Figure 4-1 
shows a normal master and slave topology with a single replication stream. 
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Master EE Slave 


Figure 4-1 MySQL master/slave replication 








Figure 4-2 shows an improved multi-master topology with two replica- 
tion streams. 


Required Multi-Master Configuration Settings 


A multi-master topology is relatively easy to set up. In addition to the con- 
figuration variables needed for standard replication (server-id and 
log-bin) there is one more mandatory configuration variable when set- 
ting up multi-master replication: 
# my.cnf 
[mysqld] 
log-slave-updates 

Used in conjunction with log-bin, this option enables a slave host to 
write activity from the SQL slave thread to its own binary log. 


Optional Multi-Master Configuration Settings 


It is highly recommended to set the passive server to read only. It is a re- 
quirement of the failover steps to ensure that read-only is set to FALSE 
at the appropriate time to enable future writes. 
# my.cnf 
[mysqld] 
read-only 

This causes the host to only accept write activity from the I/O thread or 
from users that have the SUPER privilege granted. 


| Active Master L—34 Passive Master | 


Figure 4-2 MySQL master/master replication 
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TIP A system monitoring alert check for identifying any users with SUPER 
privilege other than a predefined list will provide protection in the future. 


Other Configuration Variables to Consider 


There are other options that you may need to consider when setting up 
multi-master replication and you plan to write to both servers simultane- 
ously. The following settings are not required in an active/passive setup; 
however, if you plan to write to both hosts simultaneously you will need to 
set these. 


NOTE The following variables protect against duplication of auto-increment 
fields only. Duplicates can still occur on the server if you write the same data 
to both masters and update the same row(s). In this case, MySQL may silently 
overwrite records, leading to invalid or missing data. You can also encounter 
key clashes when UNIQUE keys are present. 


e auto increment increment Controls the interval between 
successive column values. 


e auto increment offset Determines the starting point of the 
AUTO INCREMENT column. 


e slave exec mode lfslave exec mode is IDEMPOTENT, which 
is generally only used for multi-master replication and the MySOL 
Cluster NDB storage engine, a failure to apply changes when using 
row-based replication (RBR) because the original row cannot be 
found does not trigger an error, causing replication to fail. 


CAUTION Setting s1ave exec mode to IDEMPOTENT can cause data 
drift between the master and slave. 


NOTE Ifyou have set replicate-do-dbor replicate-ignore-db, 
you will need to ensure these are set the same on both servers. 


Example Configuration 


The following outlines the recommended configuration file options for a 
MySQL pair or MySQL active/passive master setup. For this chapter we 
will use the VirtualBox environment that is defined in the appendix with 
the servers running MySQL 5.5. These server names are alpha and beta. 
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Active Server (alpha) 

On the master server, the following configuration is defined: 
[mysqld] 

server-id = 51 

log-bin = mysql-bin 

relay-log = relay-log 

read_only = FALSE 

log-slave-updates 

skip-slave-start 


Passive/Standby Server (beta) 

On the failover server, the following configuration is defined: 
[mysqld] 

server-id = 52 

log-bin = mysql-bin 

relay-log = relay-log 

read_only = TRUE 

log-slave-updates 


NOTE The slave replication that is operating on the master is disabled on initial 
server startup. This is important for certain specific error situations with 
multiple replication streams for a failed failover or unexpected server restart. This 
does require an additional management check and execution step when MySQL 
is initiated on both servers. An alternative view is to disable slave startup on both 
servers. This ensures the configuration files are more consistent between both 
servers, and then ensures a human verification step is necessary before starting 
replication. 


CAUTION These configuration settings are defined to demonstrate the setup 
and manual failover of an active/passive environment. The use of appropriate 
additional configuration to maintain durability is important for any production 
system. 


Replication Setup 


The key to this setup and failover is the configuration of replication between 
both servers. Replication only travels one way, from the master to the slave. 
When setting up multi-master there are two replication streams to set up. 
Replication runs from the active server to the passive server, where the pas- 
sive server is the active server's slave. Replication also runs from the passive 


84 Effective MySQL: Replication Techniques in Depth 


server to the active server, where the active server acts as a slave to the pas- 
sive server. This means that SHOW MASTER STATUS and SHOW SLAVE STA- 
TUS work on both servers. 


Obtain Master Status on Active Server 
Replication requires the master log file and position available from SHOW 
MASTER STATUS. This can be obtained from both servers: 


alpha> SHOW MASTER STATUS\G 
Ckckckckckckckckckckckckckckck ck ck ck k kk kkk |. row XXX kk kk kk ck ck ck kk kk kk kkkeÀ 
File: mysql-bin.000001 
Position: 107 
Binlog Do DB: 
Binlog Ignore DB: 
beta» SHOW MASTER STATUS\G 
kckckckckckckckckckckckckckck ck ck ck ck ck ck ck k kk kk |l. row kk KKK KKK ckckckckckckckckckck ck ck ck k kk 
File: mysql-bin.000001 
Position: 107 
Binlog Do DB: 
Binlog Ignore DB: 


Create Replication User 
In order for a slave server to use replication, an appropriate user has to be 
configured on each server: 


alpha» GRANT REPLICATION SLAVE ON *.* TO replGbeta IDENTIFIED BY 'repl'; 
beta» GRANT REPLICATION SLAVE ON *.* TO repl@alpha IDENTIFIED BY 'repl'; 


TIP It is recommended that when using GRANT with IDENTIFIED BY you 
first determine the hash of the password and use this value directly rather than 
a clear text password. The syntax shown here is for readability purposes and is 
not secure. 


Configure First Replication Stream 
The active and passive server can now be connected for replication with 
the CHANGE MASTER TO command: 


beta» CHANGE MASTER TO 
-» MASTER HOST-'alpha', MASTER PORT-3306, 
-» MASTER USER-'repl', MASTER PASSWORD-'repl', 
-> MASTER LOG FILE-'mysql-bin.000001', MASTER LOG POS-107, 
-» MASTER CONNECT RETRY-10; 

beta» START SLAVE; 


beta» SHOW SLAVE STATUS\G 
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LOW kk kckckckckckckckckckckckckckckckck KK kk 


Slave IO State: Waiting for master to send event 


Master Host: 

Master User: 

Master Port: 

Connect Retry: 
Master Log File: 
Read Master Log Pos: 
Relay Log File: 
Relay Log Pos: 

Relay Master Log File: 
Slave IO Running: 
Slave SQL Running: 











Master Server Id: 


Configure Second Replic 
Repeating the same steps on the a 
configure the second replication 








alpha 

repl 

3306 

10 
mysql-bin.000001 
364 
relay-log.000002 
510 
mysql-bin.000001 
Yes 

Yes 


51 


ation Stream 
ctive server, which also acts as a slave server, 
stream with CHANGE MASTER TO: 


TER_PORT=3306, 











[ER PASSWORD-'repl', 


LOG FILE-'mysql-bin.000001', MASTER LOG POS-107, 


alpha» CHANGE MASTER TO 
-» MASTER HOST-'beta', MAST 
-» MASTER USER-'repl', MAST 
-» MASTER : 
-» MASTER CONNECT RETRY-10; 
alpha» START SLAVE; 
alpha» SHOW SLAVE STATUS\G 


kckckckckckckckckckckckckckck ck ck ck ck ck k kk kk*k*k 1. 


Slave IO State: 
Master Host: 

Master User: 

Master Port: 

Connect Retry: 
Master Log File: 
Read Master Log Pos: 
Relay Log File: 
Relay Log Pos: 

Relay Master Log File: 
Slave IO Running: 
Slave SQL Running: 








Master Server Id: 


LOW *kkkckckckckckckckckckckckckckckckckckck kk kk 
Waiting for master to send event 
beta 

repl 

3306 

10 

mysql-bin.000001 

504 

relay-log.000002 

393 

mysql-bin.000001 

Yes 

Yes 


52 


Multi-master replication is now up and running. 


CAUTION With multi-master replication, the SHOW MASTER STATUS 
and SHOW SLAVE STATUS commands operate on both systems, unlike a 


traditional setup where SHOW 


MASTER STATUS displays binary log 


information, and SHOW SLAVE STATUS displays replication information 


on the slave. 
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Multi-Master Replication Verification 


A simple verification step can be used to ensure that both replication 
streams are working. That is, a DML or DDL statement on the active master 
is replicated to the slave, and a DML or DDL statement on the slave (i.e., the 
passive master) is replicated to the master. This verification is used to dem- 
onstrate and confirm operation. 

This verification only works when we use a MySQL user that has been 
granted the SUPER privilege along with CREATE, DROP, SELECT, INSERT, 
and DELETE. The SUPER privilege bypasses read only on the slave host 
and allows DML queries to run on any production system without impact. 


CAUTION Never use a user with SUPER privilege for application access to 
your data. When an application requires this privilege to manage objects, for 
example, triggers in MySQL 5.0, it is recommended that a dedicated DBA user 
with localhost only privileges is defined and used. 


First Replication Stream Verification 
On the active master server: 


alpha» CREATE SCHEMA IF NOT EXISTS verify failover; 
alpha» USE verify failover 

alpha» CREATE TABLE rpl test (id SERIAL) ENGINE - InnoDB; 
alpha» INSERT INTO rpl test(id) VALUES(1), (2),(3); 

alpha» SELECT * FROM rpl test; 








+----+ 
| id | 
*x---- 
| i| 
| 2 | 
| 3 | 
+----+ 


3 rows in set (0.01 sec) 


On the passive master server: 


beta» USE verify failover 
beta» SELECT * FROM rpl test; 


+----4 
| id | 
+----+ 
| i| 
| 2| 
|3| 
+----4 


3 rows in set (0.01 sec) 


Using Multi-Master Replication 87 


Second Replication Stream Verification 
On the passive master server: 


beta» 
beta» 
beta» 
beta» 


USE verify failover 

DELETE FROM rpl test WHERE id-2; 

INSERT INTO rpl test(id) VALUES (11), (22); 
SELECT * FROM rpl test; 





id 





11 
22 











4 row 


S in set (0.00 sec) 


On the active master server: 


alpha 
alpha 


» USE verify failover 
» SELECT * FROM rpl test; 





id 





TT 
22 








4 row 
alpha 





S in set (0.00 sec) 
> DROP SCHEMA verify failover; 


And a final verification on the passive master server, which should 


result in the error provided: 


beta> 


USE verify failover 


ERROR 1049 (42000): Unknown database 'verify failover' 


Application Usage and Verification 
The final setup step is to define an application user for normal access and 


for any further testing during failover: 


alpha> 
alpha> 
alpha> 
alpha> 
alpha> 


CREATE SCHEMA book3; 

CREATE USER app@'192.168.1.%' IDENTIFIED BY 'sakila'; 

GRANT INSERT,UPDATE,DELETE,SELECT ON book3.* TO app@'192.168.1.%'; 
USE book3 

CREATE TABLE verify (id SERIAL) ENGINE - InnoDB; 
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Manual Failover Process 


In this section we will step through the manual slave promotion process. 
This example is intended to highlight the complexities of automating this 
process. In summary: 


e Restrict write access to active master 

e Verify no write access 

e Ensure MySQL replication is up to date 
e Enable write access on failover master 


e Support application needs 


Restrict Write Access 
Stop application access to the active master server but keep MySQL run- 
ning. You accomplish this by first setting read only to TRUE and option- 
ally changing or dropping application user access or setting an appropriate 
firewall rule with iptables. 

In this case, set the active master read only variable to TRUE. This 
operation requires a user with the SUPER privilege. 


alpha» SET GLOBAL read only - TRUE; 


At this time you should also modify the default configuration of MySQL 
on the server to match the new read-only status. This is necessary to define 
the startup state if MySQL is restarted at a future time. If the active master 
also has skip-slave-start defined and this is not specified also on the 
failover master, this should also be removed to reflect the startup state of 
the passive master. 


Verify No Write Access 

Verify the active master is read only with a user that does not have the 
SUPER privilege. It is recommended that you do not run any SOL that 
would damage your current data: 


$ mysql -uapp -psakila -halpha book3 

alpha» INSERT INTO verify VALUES (NULL); 

ERROR 1290 (HY000): The MySQL server is running with the --read-only option 
So it cannot execute this statement 


Using Multi-Master Replication 89 


Ensure Replication Is Up to Date 
A verification of the master and slave status is necessary to ensure the 
slave that will be promoted to the active master has replication running 
and has completed all transactions. 

On the current active master, obtain the current master replication posi- 
tion: 


alpha» SHOW MASTER STATUS AG 


kckckckckckckckckckckckckckckckckckckckck ck k k k kk |l. row *'kkkckckckckckckckckckckckckckckck ck ck kk kk 
File: mysql-bin.000003 
Position: 26312 
Binlog Do DB: 
Binlog Ignore DB: 


On the failover slave, confirm the position matches the master: 


beta» SHOW SLAVE STATUS\G 


kckckckckckckckckckckckckckck ck ckckckckck ck ck k k kk 7l. Y OW kkckckckckckckckckckckckckckckckckckck kc kc k kk 


Master Log File: mysql-bin.000003 


Slave IO Running: Yes 
Slave SOL Running: Yes 


Exec Master Log Pos: 26312 
Master Server Id: 51 


CAUTION This step relies on the requirement that no additional data changes 
are occurring on the master. The read-only status can be easily overridden by 
any user that has the SUPER privilege. This is another primary reason why an 
application user should never have this privilege. 


Enable Write Access on New Master 

At this time, the current master is defined as read-only and replication is 
confirmed as operational. A sanity check is shown to indicate the new 
failover master is not currently supporting writes: 


$ mysql -uapp -psakila -hbeta book3 

beta» INSERT INTO verify VALUES (NULL); 

ERROR 1290 (HY000): The MySQL server is running with the --read-only option 
So it cannot execute this statement 
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CAUTION If the new failover master fails the read-only check, failover may 
still continue. However, this is an indication that data inconsistency is 
possible. Having an appropriate check of data consistency via checksums is 
an important monitoring step in a multi-master environment. 


Using a user with the SUPER privilege, the slave that is now the new 
active master is enabled to accept writes: 
beta> SET GLOBAL read_only = FALSE; 
$ mysql -uapp -psakila -hbeta book3 

A final sanity check with an application user with appropriate privileges 
can be performed, i.e., the earlier check that failed due to the read-only state: 
beta» INSERT INTO verify VALUES (NULL); 

Query OK, 1 row affected (0.01 sec) 

Additional steps, for example, enabling suitable application user per- 
missions that are disabled by default, or removing a firewall restriction, 
may also be needed depending on what additional steps are required in 
the environment. 


TIP Additional care has to be taken when adding and removing MySQL user 
privileges, as the replication of the mysq1 schema would cause this to be 
replicated also. Certain commands may need to be excluded from binary logging. 


Verifying Resumed Operations 

A final check should be performed with transaction throughput on the new 
active master to ensure the second replication stream is operating as expected. 
A SHOW PROCESSLIST will confirm statements are being executed on the 
new active master. It is important that the SHOW MASTER STATUS and 
SHOW SLAVE STATUS is performed on the correct servers, i.e., the inverse of 
what was performed before commencing the manual failover: 


beta» SHOW MASTER STATUS\G 
kckckckckckckckckckckckckckckckckckck ck ck ck k k k kk |l. row *'kkkckckck ck ckckckckckckckckckck ck ck ck ck k kk 
File: mysql-bin.000001 
Position: 33426 
Binlog Do DB: 
Binlog Ignore DB: 


alpha > SHOW SLAVE STATUS\G 
kckckckckckckckckckckckckckckck ck ck ck kc kc kc ck k k kk 7. row BERR KKK KKK KEKE KEK KKK ckckckckckc kc kc k kk 
Slave IO State: Waiting for master to send event 
Master Host: beta 
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Master User: repl 
Master Port: 3306 
Connect Retry: 10 
Master Log File: mysql-bin.000001 
Read Master Log Pos: 33426 
Relay Log File: relay-10g.000002 
Relay Log Pos: 1044 
Relay Master Log File: mysql-bin.000001 
Slave IO Running: Yes 
Slave SOL Running: Yes 








Exec Master Log Pos: 33426 


Master Server Id: 52 
1 row in set (0.00 sec) 


CAUTION The location of the Read Master Log Pos column in the 
SHOW SLAVE STATUS output should not be mistaken for the need to verify 
the Exec Master Log Pos that occurs later in the output. 


NOTE It is very easy to be confused when running SHOW MASTER STATUS 
and SHOW SLAVE STATUS on the wrong servers in a MySQL pair 
configuration. Both servers act as a master and a slave in a replication sense. 


Manage Application Access 

In these examples, there is a clear manual understanding of which server 
is the active master and which server becomes the new active master. In an 
application situation, that must be pre-determined or managed in addition 
to the steps just highlighted to minimize application downtime. A common 
approach is to use a virtual IP (VIP) address that may optionally resolve to 
a common DNS name. The application configuration always communicates 
with the VIP or common name rather than the physical IP address of the 
active master server. The failover approach then manages the changing of 
the VIP to point to the correct server that is acting as an active master. For 
example, add a VIP address to an existing physical network adapter on the 
master: 


alpha$ sudo ifconfig eth1:0 192.168.1.101 up 
alphas ifconfig 


eth1 Link encap:Ethernet  HWaddr 08:00:27:d4:5f:91 
inet addr:192.168.1.51 Bcast:192.168.1.255 Mask:255.255.255.0 
inet6 addr: fe80::a00:27f£f:fed4:5£91/64 Scope:Link 
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 
RX packets:370474 errors:0 dropped:0 overruns:0 frame:0 
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TX packets:1184045 errors:0 dropped:0 overruns:0 carrier:0 
collisions:0 txqueuelen:1000 
RX bytes:31804703 (31.8 MB) TX bytes:1671393644 (1.6 GB) 


eth1:0 Link encap:Ethernet  HWaddr 08:00:27:d4:5f:91 
inet addr:192.168.1.101 Bcast:192.168.1.255 Mask:255.255.255.0 
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 


In addition, this should be added to the /etc/network/interfaces 
file, or applicable file for your operating system, to preserve this state when 
the system is restarted. 

The application is modified to always connect to this IP address or a DNS 
entry. The use of an IP address for database connections removes additional 
DNS translation overhead and possible DNS server lookup failures. For this 
example the new host information is defined on all servers in the test 
environment: 

#/etc/hosts 
192.168.1.101 master 

When performing a failover, the removal of the IP address from the 
master should be performed when the server is set to read-only: 


alpha$ sudo ifconfig eth1:0 down 


When read-only access is removed on the failover master, the IP address 
should then be enabled on the new active master: 
beta$ sudo ifconfig eth1:0 192.168.1.101 up 
betas ifconfig 

The application will operate with this VIP because the application user 
privileges were defined for a wildcard host, i.e., 192.168.1.% 

A further complication with this network mapping is you may need to 
have a network admin run an Address Resolution Protocol (ARP) cache 
clear to ensure the passive slave’s Media Access Control (MAC) address 
will be assigned to the VIP. ARP is primarily used to connect the OSI Mod- 
el Network Layer (Layer 3) to the Data Link Layer (Layer 2). For most net- 
works this refers to IP and MAC mapping.The arp and arping commands 
can be used in identification and notification steps. This link provides a 
good introduction to these commands: http://homepage.smc.edu/morgan_ 
david/cs75/labs/arp-and-arping.htm. 
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Review 

The manual slave promotion process has a lot of steps and is prone to human 
error. Floating IP addresses or VIPs add a level of complexity in that a net- 
work administrator will need to be involved, plus a MySQL administrator 
may not have access to run the IP removal and addition to the servers, pos- 
sibly needing a system administrator to run those commands. 

Certain cloud infrastructures may not be able to assign a second IP ad- 
dress that can float from server to server. Until very recently, it was not pos- 
sible to have a second IP address with Amazon Web Services (AWS). This 
was announced on July 6, 2012 (see http://aws.typepad.com/aws/2012/07/ 
multiple-ip-addresses-for-ec2-instances-in-a-virtual-private-cloud.html). 
This requires a different approach to steps 6, 7, and 8 earlier, which may 
include changing a connection string in your application and restarting 
your web server. 


Real World Usage Complications 


While these steps hint at a plausible solution, the reality is there are many 
additional situations where this process has potential flaws or additional 
complications. In a fully controlled situation, these initial steps do provide a 
viable solution. Some of the complexities that have to be considered include 


e Management or administrative processes that are executed on the 
master, for example, batch processes. These should be stopped 
before a controlled failover. These processes may require additional 
configuration if they are designed to run on the local master server. 


e The unavailability of the actual master server does not enable a 
controlled failover. There is no guarantee MySQL replication is up to 
date, and the managed state of changing the master read-only at 
runtime and configuration is not possible. If a VIP is in use, this is not 
actively removed. This situation can result in a split-brain where 
both masters could receive data. This is a disaster situation that can 
cause data corruption. 


e The use of persistent connections, for example, Connector/J, can add 
additional complexities to connection management. Additional 
testing and error validation are necessary to ensure the application 
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supports the physical change of the underlying server that is 
servicing requests with the VIP. It is recommended that a flush of 
persistent connection pool on application servers is performed to 
minimize complexities during the failover. 


Long running queries on the master may result in additional 
complications. While the setting of read-only will address new 
writes, including within a transaction, any long running queries 
should be monitored before completing a failover. 


In an uncontrolled failover situation the write consistency, especially 
of slave information, may cause replication issues. Additional 
configuration settings may be necessary to ensure file 
synchronization of important replication information; however, this 
has a performance impact for all transactions. The specific issue of 
file consistency of replication slave information has been addressed 
with crash-consistent slaves available in MySQL 5.6. 


A management server or arbitrator can be used to manage split- 
brain or other network unavailability situations. When the MySOL 
pair is initiated, each server asks an arbitrator (i.e., a third party) who 
is the master rather than starting with a default state. 


Unless continuously checksummed, there is no absolute 
confirmation that data on both servers is synchronized. 


Additional Slave Servers 


While this initial example used two MySQL servers, it is possible to have ad- 
ditional replication slaves in the MySQL topology. The use of a MySQL pair, 
i.e, two servers to support writes and additional slaves to support reads, can 


easily be supported with the described multi-master replication environment. 


These additional servers can be managed with: 


e Replication slaves connected with the active master. 


e Replication slaves moved from the active master to the new master 


before the failover occurs. 


In the example, by adding a third server using the virtual environment 


defined in the appendix, these steps can be demonstrated. 
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Slave Server Setup (gamma) 
On a new slave server, the following configuration is defined: 


[mysqld] 

server-id = 53 
relay-log = relay-log 
read_only=TRUE 


The setup of the slave repeats the same steps for the initial replication 
streams. First create an appropriate replication user: 


alpha» GRANT REPLICATION SLAVE ON *.* TO repl@gamma 
IDENTIFIED BY 'repl'; 


Initiate replication on the new slave: 


gamma» CHANGE MASTER TO 
-» MASTER HOST-'alpha', MASTER PORT-3306, 
-» MASTER USER-'repl', MASTER PASSWORD-'repl', 
-» MASTER LOG FILE-'mysql-bin.000001', MASTER LOG POS-107, 
-» MASTER CONNECT RETRY-10; 

gamma» START SLAVE; 

gamma» DO SLEEP(2); 

gamma» SHOW SLAVE STATUSNG 














The MySQL topology now has three MySQL servers, a master server 
(alpha), a failover master (beta), and an additional slave (gamma). Figure 4-3 
shows the servers and the replication streams currently configured. This 
figure includes an indication that additional slave servers can be added in 
the same fashion. 
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Figure 4-3 MySQL master/master replication with additional slaves 
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It is also possible to split slaves between both the active and passive 
master and not move slaves during the failover. This has the added benefit 
that in an uncontrolled failure half the slaves are always in operation. This 
also has the added risk that with certain replication failure situations, half 
the slaves are out of date. 


Slave Server Management 

As described in the introduction, the key to using additional slaves effec- 
tively is for the slaves to connect to the active master. In a failover situation 
the slave should be moved to the new master prior to failover. This involves 
the following steps: 


1. Ensure slave is operating and up to date. 


2. Obtain a consistent view of both replication streams on the 
failover server. 


Ensure Slave Operation 


gamma» SHOW SLAVE STATUS AG 


kckckckckckckckckckckckckckck ck ck ck ck ck k k k kk kk 7]. 
Slave IO State: 

Master Host: 

Master User: 

Master Port: 

Connect Retry: 
Master Log File: 


Slave IO Running: 
Slave SQL Running: 


Exec Master Log Pos: 
Seconds Behind Master: 
Master Server Id: 


gamma» STOP SLAVE SQL THREAD; 


LOW kk kckckckckckckckckckckckckckckckck ck ck KEE 
Waiting for master to send event 
alpha 

repl 

3306 

10 

mysql-bin.000003 


51 


Prior to stopping the SQL thread, ensure the slave is running (Slave_ 
IO Running and Slave SQL Running) and slave is up to date (Seconds _ 
Behind Master) and confirm the current master (Master Host). 


Determine Failover Position On the master you are failing over to (i.e., 
the current passive master and not the host described in the SHOW SLAVE 
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STATUS output), obtain a consistent snapshot of the slave replication 
stream and the master replication stream on the instance. For example: 











beta» STOP SLAVE SQL THREAD; 
beta» SHOW SLAVE STATUS\G 
beta» SHOW MASTER STATUSNG 
beta» START SLAVE SQL THREAD; 


With the SHOW SLAVE STATUS output you have the Master Log 
Fileandthe Exec Master Log Pos.For example: 


beta» SHOW SLAVE STATUS\G 
Master Log File: mysql-bin.000003 


Slave_IO Running: Yes 
Slave SOL Running: No 


Exec Master Log Pos: 26661 


This is used to construct the SQL statement. 


START SLAVE UNTIL 
MASTER LOG FILE-'mysql-bin.000003', MASTER LOG POS-26661; 


With the SHOW MASTER STATUS OUTPUT you have to use the new 
File and Position. For example: 


beta> SHOW MASTER STATUS\G 


Ckckckckckckckckckckckckckckck ck ck ck ck ck ck ck k kk kk |l. row BERR kckckck ck ckckckckckckck KKK ck ck ck ck k kk 
File: mysql-bin.000001 
Position: 33566 
Binlog Do DB: 
Binlog Ignore DB: 


You can use this to create the following SQL statement: 


CHANGE MASTER TO 
MASTER LOG FILE-'mysql-bin.000001', MASTER LOG POS-33566; 
Unfortunately, this is insufficient access to information to obtain the con- 
nection details of the new master. There is no easy way to obtain this infor- 
mation in one step. Some information can be obtained and inferred from 
the SHOW SLAVE STATUS, for example, the Master Userand Master _ 
Port, providing these are consistent across your topology. Also required 
are the host and the password. The host can be obtained with the following 
SQL statement in MySQL 5.1 or better: 


beta» SELECT variable value AS host name 
-» FROM information schema.global variables 
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-» WHERE variable name-'hostname'; 








This still leaves the required replication user password. There are two 
options, either use a known value for the password (i.e., you hardcode this 
in your subsequent SOL statement), or find all the information and the 
password by using the second replication stream on the current master in 
the master.info file found in the MySQL data directory: 
$ cd /path/to/datadir 
$ head -7 master.info | tail -4 
beta 
repl 
repl 
3306 

In plain text you now have all the new master connection details from 
the existing replication stream that is the slave on the new failover master. 
This information is used to construct the full SOL statement necessary: 
CHANGE MASTER TO 


MASTER HOST-'beta', MASTER USER-'repl', MASTER PASSWORD-'repl', 
MASTER LOG FILE-'mysql-bin.000001', MASTER LOG POS-33566; 





NOTE In MySQL 5.6 when using crash-safe slaves, the information from the 
master.info file is available in the mysq1.sl1ave master info table. 


Perform Slave Failover We now execute on the attached slave that is being 
moved the following SOL to align the replication stream with the recorded 
failover position: 


gamma» START SLAVE UNTIL 

-» MASTER LOG FILE-'mysql-bin.000003', MASTER LOG POS-26661; 
gamma» DO SLEEP(1); 
gamma» SHOW SLAVE STATUS AG 


kckckckckckckckckckckckckckckck ck ck ck kc kc kc ck k k kk 7. Y OW *kkckckckckckckckckckckckck ck ckckckckckc kc kc k kk 


Master Log File: mysql-bin.000003 


Slave IO Running: Yes 
Slave SOL Running: No 


Exec Master Log Pos: 26661 
Until Condition: Master 
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Until Log File: mysql-bin.000003 
Until Log Pos: 26661 


A confirmation should be performed to ensure the slave is at the correct 
location by comparing the Master Log File + Exec Master Log 
Posand Until Log File + Until Log Pos values. 

The final step is to reset the slave and point to the current master posi- 
tion of the failover master: 


gamma» STOP SLAVE; 

gamma» RESET SLAVE; 

gamma» CHANGE MASTER TO 
-» MASTER HOST-'beta', MASTER USER-'repl', MASTER PASSWORD-'repl', 
-» MASTER LOG FILE-' mysql-bin.000001', MASTER LOG POS-33566; 

gamma» START SLAVE; 

gamma» DO SLEEP(1); 

gamma» SHOW SLAVE STATUSNG 

















kckckckckckckckckckckckckckck ck ck ck ck ck ck ck k kk kk |l. row kk KKK KKK ckckckckckck ck ckck ck ck ck kk kk 


Slave IO State: Waiting for master to send event 
Master Host: beta 
Master User: repl 
Master Port: 3306 

Connect Retry: 10 


Slave IO Running: Yes 
Slave SQL Running: Yes 


Master Server Id: 52 
1 row in set (0.00 sec) 

These steps need to be repeated for each attached slave. In an environ- 
ment when you have a larger number of slaves, the chaining of the slaves 
via a relay server (aka just another slave) enables you to move just one 
slave (i.e., the relay slave) to achieve the same result. Figure 4-4 shows an 
example replication topology between the servers for this situation. 


TIP When using a large number of slaves with a relay slave, the use of the 
BLACKHOLE storage engine on the relay slave can improve replication 
performance. This minimizes the writes of the actual data, as the BLACKHOLE 
engine simply discards the data. What is necessary is the binary log on the 
relay that is used for replication with all attached slaves. 


CAUTION You should only use the BLACKHOLE storage engine with the 
technique described above when binlog-format is set to STATEMENT. 
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Active Master Passive Master 
l Relay Slave | 


Figure 4-4 MySQL master/master replication with relay slave 

















Read and Write Load Balancing 


The implementation of an active/passive multi-master environment gen- 
erally uses a virtual IP (VIP) to support the management of writes. In an 
active/active configuration, the application will need to make the decision 
of which master to write to, and balancing these writes may be an impor- 
tant consideration to manage load. The use of additional slave servers for 
read scalability also requires the application to support splitting reads and 
writes, and also balancing reads between a pool of read slaves if applicable. 
Chapter 3 described several techniques for managing these requirements. 


Circular Replication 


A MySQL pair is the use of two MySQL instances in a multi-master replica- 
tion configuration. While the recommendation is to use an active/passive 
implementation, the same approach is used for an active/active implemen- 
tation. As previously mentioned, additional configuration settings, for 
example, when using auto increment columns, is critical. MySQL replication 
does not support collision detection, so depending on the complexity of the 
application using the active/active environment, additional tricks are needed 
to minimize conditions that may cause replication to break. 
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Figure 4-5 MySQL master/master circular replication 


MySQL can support more than two servers via a circular replication 
configuration. Again, while not recommended this is indeed possible. 
Figure 4-5 shows how replication may be configured in this situation. 


NOTE Amore complex MySQL replication topology is not due to the 
configuration and effective use. The complexity and significant difficulties are 
in managing an environment after the result of some replication failure, which 
can then cause cascading replication failures or data corruption that can be 
extremely difficult to plan for and even correct. The need to consider a more 
complex topology for a more highly available replication environment indicates 
the importance of the system, and the need for high uptime should be carefully 
weighted with the increased risk of the inherent dangers that can occur. 


Other Replication Topologies 


There are many other types of MySQL replication topologies in addition to 
the simple, multi-master, and circular replication approaches that have 
been described. A tree or hierarchical architecture can produce a fan-out 
environment to support massive read scalability. MySOL replication can 
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work remarkably well with 100 to 200 MySQL slaves attached to a master. 
When the number of slaves continues to increase, for example, 300 or more, 
a multi-level tree architecture is definitely needed for performance as well 
as redundancy for any disaster situation. 

Other approaches include a hybrid circular and multi-master configura- 
tion, and complex systems that include sharding and using several different 
topologies all with one environment. 

The current MySQL replication implementation does not support one 
significant type of topology. That is a slave instance that supports writes 
from multiple masters. We will discuss this in Chapter 6 with a Tungsten 
Replicator that does provide this feature. 


Automating High Availability Failovers 


From these steps you can determine there are many moving parts to a suc- 
cessful multi-master replication environment, and there is complexity in 
ensuring all operations are performed and verified as expected. 

The following open source utilities exist in the MySOL ecosystem for 
automating failover architectures: 


e MHA for MySQL: Master High Availability Manager and tools for 
MySQL was created by Yoshinori Matsunobu and is a viable way to 
automate master failover and slave promotion. 

e MMM or Multi-Master Replication Manager for MySQL is another 
tool that monitors and automates management and failover of 
multi-master MySOL clusters. 


Flipper is another legacy tool that supports managing a simple 
MySQL pair configuration. 

More information about these tools is available in Chapter 5. 

Starting with MySOL 5.6, many of these steps and additional utilities 
can be simplified or eliminated with the introduction of features including 
UUID, GTID, and crash-safe slaves that were discussed in Chapter 3. In 
Chapter 5 several utilities leveraging these new features are also discussed. 
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Conclusion 


Running multi-master topologies with native MySOL replication has been 
around for a long time. Multi-master replication is not complex to configure; 
however, it is complex to manage and support for all production situations 
and possible disaster recovery needs. Successful implementations using 
multi-master replication generally involve designing the application to be 
aware of, and cater for, the inherent limitations and risks. 

Examples and links in this chapter are available for download from 
http://EffectiveMySQL.com/book/replication-techniques. 
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MySQL Replication Tools 


The MySQL ecosystem includes many different tools that can be used to 
support, manage, and monitor MySQL replication. These tools have been 
developed to enhance the default MySOL installation. Over time MySOL 
has incorporated some of these community features and tools. 

In this chapter we discuss: 


e Various available toolkits, including Openark Kit, Percona Toolkit, 
Maatkit, and MySOL Workbench Utilities 


e Replication prefetch options 


e MySQL failover managers, including MySQL MHA, MMM, and Flipper 
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Various MySOL Toolkits 


Several individuals and companies have created a number of tools and 
combined them into various toolkits. The most common toolkits and the 
relevant MySQL replication utilities are included. Each of the examples in 
the following toolkits relies on the configuration of a suitable MySOL 
topology. The MySQL Sandbox and/or the VirtualBox setup that is defined 
in the appendix is used. 


Openark Kit 


Created and maintained by the 2009 MySOL community member of the 
year and Oracle ACE Shlomi Noach (http://code.openark.org), Openark 
provides a number of common utilities to administer, diagnose, and audit 
MySQL databases.The following are the related MySOL replication utilities. 

This software is written in Python and is available under the open source 
BSD license. More information can be found at http://code.openark.org/ 
forge/openark-kit. 


Installation 
The following steps install Openark and the necessary dependencies on an 
Ubuntu/Debian server: 


$ cd /tmp 
$ wget http://openarkkit.googlecode.com/files/openark-kit-180-1.deb 
$ sudo apt-get install python-mysqldb 
$ sudo dpkg -i openark-kit-*.deb 
$ rm -f openark-kit-*.deb 
$ oak-get-slave-lag -help 
The following steps install Openark and the necessary dependencies on 
a Red Hat/CentOS/Oracle Linux server: 
$ cd /tmp 
$ wget http://openarkkit.googlecode.com/files/openark-kit-180-1.noarch.rpm 
$ sudo yum install MySQL-python 
$ sudo rpm -i openark-kit-*.noarch.rpm 
$ rm -f openark-kit-*.noarch.rpm 
$ oak-get-slave-lag -help 


Refer to http://code.google.com/p/openarkkit/downloads/list for de- 
tails of the most current version and software for other operating systems. 
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oak-get-slave-lag 

The oak-get - s1ave- 1ag utility provides a convenience script of capturing 
the Seconds Behind Master information from SHOW SLAVE STATUS. 
This also supports an acceptable amount before providing an error. The 
following examples use a MySQL Sandbox environment as defined in the 
appendix: 


$ cd SHOME/sandboxes/rsandbox 5 5 24 


$ oak-get-slave-lag --defaults-file-nodel/my.sandbox.cnf 
0 

In an error situation when replication lag exceeds a given number of 
seconds specified: 


$ oak-get-slave-lag --defaults-file-nodel/my.sandbox.cnf -e 5 
-- ERROR: 8 


When replication is not running: 


$ ./sl -e "STOP SLAVE" 

$ oak-get-slave-lag --defaults-file-nodel/my.sandbox.cnf 
None 

$ ./sl -e "START SLAVE 


References 
Code: http://openarkkit.googlecode.com/svn/trunk/openarkkit/doc/html/ 
oak-get-slave-lag.html. 


oak-show-replication-status 

This command provides a convenient report of the MySOL topology for a 
given master. This will automatically detect and check all slaves for the spec- 
ified master. This will not report slaves in a nested topology. For example: 


$ cd S$HOME/sandboxes/rsandbox 5 5 24 

$ oak-show-replication-status --defaults-file=master/my.sandbox.cnf 
-- master log: mysql-bin.000003 

-- Slave host Slave port Master Log File Seconds Behind Master 
Status 
-- ERROR: Cannot SHOW SLAVE STATUS on SBslavel:21380 


-- ERROR: Cannot SHOW SLAVE STATUS on SBslave2:21381 

















In this situation you can see errors. This is due to the underlying process of 
using SHOW SLAVE HOSTS, which requires the - - report -host option to 
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be defined on slaves. By adding SBslave1 and SBslave2 to your machine 
hosts file, a correct output is provided to demonstrate the utility: 


$ oak-show-replication-status --defaults-file-master/my.sandbox.cnf 
-- master log: mysql-bin.000001 


-- Slave host Slave port Master Log File Seconds Behind Master Status 
-- SBslavel 21380 mysql-bin.000001 0 Good 
-- SBslave2 21381 mysql-bin.000001 0 Good 


CAUTION There is no validation in MySQL for the - -report - host value 
as shown with using MySQL Sandbox. If this is not an actual server name, 
this utility will not produce the results you may be expecting. 


When using the virtual environment as defined in the appendix: 


$ oak-show-replication-status -uroot -ppasswd -Halpha 

-- master log: alpha-bin.000007 

-- Slave host Slave port Master Log File Seconds Behind Master Status 

-- beta 3306 alpha-bin.000007 0 Good 
oak-show-replication-status does not support nested master/ 

slave topologies as demonstrated in the later MySQL Workbench Utilities 


example. 


References 
Code: http://openarkkit.googlecode.com/svn/trunk/openarkkit/doc/html/ 
oak-show-replication-status.html. 


oak-purge-master-logs 

The master binary logs require monitoring and management to ensure 

available disk space exists on the master server for normal database opera- 

tions. The expire logs days configuration option and the PURGE 

MASTER LOGS command can be used to maintain a healthy amount of 

files. While expire 1ogs days will only delete binary logs older than 

the specific amount when a new binary log is created, PURGE MASTER 

LOGS can remove files by name or to a certain date and time. In addition 

the oak-purge-master-logs command can perform removal of binary 

logs for an applicable retention policy with the number of available files. 

For example: 

$ cd $HOME/sandboxes/rsandbox 5 5 24 

$ oak-purge-master-logs --retain-logs-3 --print-only \ 
--defaults-file-master/my.sandbox.cnf 

PURGE MASTER LOGS TO 'mysql-bin.000005' 
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This syntax will provide what can be run manually. To confirm the cor- 
rect information you can review the existing binary logs in place with: 


$ ls -l1 master/data/mysql-bin* 


-rw-rw---- 1 uid gid6309531 May 21 22:10 master/data/mysql-bin.000001 
-rw-rw---- 1 uid gid 5327 Jun 7 09:26 master/data/mysql-bin.000002 
-rw-rw---- 1 uid gid 2981 Jun 12 18:42 master/data/mysql-bin.000003 
-rw-rw---- 1 uid gid 126 Jun 12 18:50 master/data/mysql-bin.000004 
-rw-rw---- 1 uid gid 6354 Jun 14 12:18 master/data/mysql-bin.000005 
-rw-rw---- 1 uid gid 27650 Jun 14 13:32 master/data/mysql-bin.000006 
-rw-rw---- 1 uid gid 107 Jun 14 13:32 master/data/mysql-bin.000007 
-rw-rw---- 1 uid gid 133 Jun 14 13:32 master/data/mysql-bin.index 
$ oak-purge-master-logs --retain-logs=3 \ 


--defaults-file=master/my.sandbox.cnf 
$ ls -l1 master/data/mysql-bin* 


-rw-rw---- 1 uid gid 6354 Jun 14 12:18 master/data/mysql-bin.000005 
-rw-rw---- 1 uid gid 27650 Jun 14 13:32 master/data/mysql-bin.000006 
-rw-rw---- 1 uid gid 107 Jun 14 13:32 master/data/mysql-bin.000007 
-rw-rw---- 1 uid gid 133 Jun 14 13:32 master/data/mysql-bin. index 


This utility can also ensure that binary logs are maintained for any slaves 
that may be out of date. 


CAUTION It is important you keep the current binary logs from the last static 
backup available. This should be part of your standard backup and recovery 
process. If your system writes a large number of files per day, for example, 100 
files, removing files by a constant number without an adequate backup may 
delete important binary logs for a disaster recovery from the previous static 
backup or an older static backup. These are important business considerations 
to review in addition to the technical needs. 


The Effective MySQL: Backup and Recovery book (McGraw-Hill, 2012) pro- 
vides extensive details on how to manage the MySQL binary logs, includ- 
ing how to make copies, and how to monitor and verify copied files. 


References 
Code: http://openarkkit.googlecode.com/svn/trunk/openarkkit/doc/html/ 
oak-purge-master-logs.html. 


Percona Toolkit 


The Percona toolkit, by the product name owners, provides various utilities 
including the following commands relevant to MySQL replication. This 
software is available under the GPL v2 license and can be obtained from 
http://www.percona.com/software/percona-toolkit/. 
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Installation 
The following instructions install Percona Toolkit on an Ubuntu/Debian 


distribution: 
$ wget http://www.percona.com/redir/downloads/percona-toolkit/2.1.1/ 
percona-toolkit 2.1.1 all.deb 
$ sudo apt-get install libterm-readkey-perl 
$ sudo dpkg -i percona-toolkit 2.1.1 all.deb 
$ pt-slave-find -help 
The following instructions install Percona Toolkit on a Red Hat/CentOS/ 
Oracle Linux distribution: 
$ wget http://www.percona.com/redir/downloads/percona-toolkit/2.1.1/ 
percona-toolkit-2.1.1-1.noarch.rpm 
$ sudo yum install perl-TermReadKey 
$ sudo yum install perl-DBD-MySQL 
$ sudo rpm -i percona-toolkit-2.1.1-1.noarch.rpm 
$ pt-slave-find -help 
Refer to http://www.percona.com/downloads/percona-toolkit/ LATEST/ 
for the most current version and software for other operating systems and 


distributions. 


pt-table-checksum 

One feature that is essential for managing and ensuring replication data 
consistency is appropriate table checksum verification. The Percona Toolkit 
provides the pt -table-checksum command, which replaces the Maatkit 
mk-table-checksum tool. The following example shows a test table 
that is missing a row on the slave using the MySOL Sandbox environment 
that is defined in the appendix: 


CAUTION This utility is disk intensive for large databases as by default this 
reads the entire table for the specified schemas and tables. 


$ cd SHOME/sandboxes/rsandbox 5 5 24 

$ ./m 

master» CREATE SCHEMA IF NOT EXISTS book3; 

master» USE book3 

master» CREATE TABLE difference test (id INT NOT NULL); 
master» INSERT INTO difference test VALUES (1),(2),(3); 
$ ./sl 

Slave» USE book3 

Slave» DELETE FROM difference test LIMIT 1; 
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Running the pt -table-checksum command: 


$ pt-table-checksum --databases-book3 --defaults-file-master/my.sandbox.cnf \ 
--replicate book3.checksum 


TS ERRORS DIFFS ROWS  CHUNKS SKIPPED TIME TABLE 
06-14T15:57:31 0 0 3 1 0 0.031 book3.difference test 
06-14T15:57:31 0 0 1 1 0 0.028 book3.no tbl on slave 
06-14T15:57:32 0 0 1048576 7 0 1.148 book3.numbers 
06-14T15:57:32 0 0 2 1 0 0.030 book3.rbr_test 
06-14T15:57:32 0 0 4 1 0 0.027 book3.uniq test 


Full details of the checksums between the master and slave tables can 
be found in the book3 . checksum table. The following SOL will provide a 
difference recorded in this table if any exists: 


Slave» SELECT db, tbl, SUM(this cnt) AS total rows, 
-» COUNT(*) AS chunks 
-» FROM book3.checksum 
-> WHERE ( master cnt <> this cnt OR 
-> master crc <> this crc OR 
-> ISNULL(master crc) <> ISNULL(this crc)) 
-> GROUP BY db, tbl; 

















1 row in set (0.00 sec) 


For comparison, the second slave shows no differences: 


$ ./s2 
Slave» SELECT 
Empty set (0.01 sec) 


If there are missing schema objects in your slave schema, this command 
will break replication. For example: 


$ ./51 
Slave» SHOW SLAVE STATUS\G 
kkkkkkkkkkkkkkkkkkkkkkkkkkk 1 . row koc ck ce cec check ce ck ce ck ce kc ke ck ke ck ce ck ck kk kk kx 


Slave IO Running: Yes 
Slave SOL Running: No 


Last Errno: 1146 
Last Error: 'Table 'book3.no tbl on slave' doesn't exist' 
on query. Default database:'book3'.Query: 'REPLACE INTO ^book3^.^checksum^ (db, 
tbl, chunk, chunk index, lower boundary, upper boundary, this cnt, this crc) 
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SELECT 'book3', 'no tbl on slave', '1', NULL, NULL, NULL, COUNT(*) AS cnt, 
COALESCE(LOWER(CONV(BIT XOR(CAST(CRC32(^id^) AS UNSIGNED)), 10, 16)), 0) AS crc 
FROM ^book3^.^no tbl on slave^ /*checksum table*/' 


The following utility describes how to identify the actual row differences. 


References 
Documentation: http://www.percona.com/doc/percona-toolkit/pt-table- 
checksum.html. 


pt-table-sync 
This tool attempts to identify data changes and synchronize these via 
re-applying commands on the MySQL master. This tool does change data 
and comes with a recommendation that you backup your data before use. 
You can also run the tool in print-only mode to see what operations would 
be run. This is generally used in conjunction with an identified data drift 
from the output of pt -table-checksum. 

Demonstrating how to identify the difference of the slave data as 
described with the pt -table-checksum command: 


4 This is needed for the full connection string required for this utility 

$ cat nodel/my.sandbox.cnf 

$ pt-table-sync --print --sync-to-master --databases-book3 h-localhost,P-21380, 
u=msandbox, p=msandbox, S=/tmp/mysql_sandbox21380.sock 


REPLACE INTO ^book3^."checksum  (^db^, ^tbl^, ^chunk^, ^chunk time", 
^chunk index^, "lower boundary, "upper boundary^, ^this crc^, ^this cnt', 
^master crc^, “master cnt^, ^ts^) VALUES ('book3', 'difference test', '1', 
'0.008657', NULL, NULL, NULL, 'f4dbdf21', '3', 'f4dbdf21', '3', '2012-06-14 
15:57:31'); 
REPLACE INTO ~book3~. difference test'(^id^) VALUES ('1') 

/*percona-toolkit src db:book3 src tbl:difference test ... */; 


Table book3.no tbl on slave does not exist on P-21380,S-/tmp/mysqgl sandbox21380. 
sock,h=localhost,p=...,u=msandbox 
while doing book3.no tbl on slave on localhost 


As you can see from the output, more information was provided than 
expected. This utility does identify the data drift in the difference table 
and provides a suitable REPLACE command that, if run on the master, 
would provide for consistent data in this table. What is also shown are mod- 
ifications to synchronize the actual checksum table that you do not want to 
be modified. The utility also highlights a difference of a missing table. 


CAUTION While this tool can be used to identify physical data differences, it 
is recommended that all SQL statements are carefully verified before execution 
to ensure no data loss. 
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References 
Documentation: http://www.percona.com/doc/percona-toolkit/pt-table- 
sync.html. 


pt-heartbeat 

This utility attempts to monitor replication delay by monitoring actual rep- 
licated information. This is performed in two parts, the - -update com- 
mand, and the - -check or -- monitor command: 


$ cd S$HOME/sandboxes/rsandbox 5 5 24 


$ pt-heartbeat --defaults-file-master/my.sandbox.cnf \ 
--ereate-table --databasesbook3 --tablesheartbeat --update & 


You can review what is actually being created with: 


$ ./m 
master» SELECT * FROM book3.heartbeat\G 
kckckckckckckckckckckckckckck ck ck ck ck ck ck ck k k k kk |l. row **'kkkckckckckckckckckckckckckckckck ck kk k kk 
ts: 2012-05-21T17:19:20.001820 
Server id: 1 
file: mysql-bin.000001 
position: 231760 

relay master log file: NULL 

exec master log pos: NULL 
1 row in set (0.00 sec) 














master» SELECT * FROM book3.heartbeat\G 
kckckckckckckckckckckckckckck ck ck ck ck ck ck ck k kk kk |l. row kk KKK KKK ckckckckckckckckckck ck ck kk kk 
ts: 2012-05-21T17:19:21.001820 
Server id: 1 
file: mysql-bin.000001 
position: 232097 
relay master log file: NULL 
exec master log pos: NULL 
1 row in set (0.00 sec) 














Replication is monitored with: 


$ pt-heartbeat --defaults-file-master/my.sandbox.cnf \ 
--database-book3 --master-server-id 1 --check 


0.00 


or 


$ pt-heartbeat --defaults-file-master/my.sandbox.cnf \ 


--database-book3 --master-server-id 1 --monitor 
0.00s [ 0.00s, 0.00s, 0.00s ] 
0.00s [ 0.00s, 0.00s, 0.00s ] 
0.00s [ 0.00s, 0.00s, 0.00s ] 
0.00s [ 0.00s, 0.00s, 0.00s ] 
0.00s [ 0.00s, 0.00s, 0.00s ] 


114 Effective MySQL: Replication Techniques in Depth 


References 
Documentation: http://www.percona.com/doc/percona-toolkit/pt-heart- 
beat.html. 


pt-slave-delay 

This utility allows a MySQL slave to have a certain delay behind the MySQL 
master. In some environments this feature can offer access to data that may 
be modified or deleted accidently on a master.This is not a replacement for 
a backup if data is deleted; however, it can provide a more convenient view 
of the data in question. 


$ cd SHOME/sandboxes/rsandbox 5 5 24 








$ pt-slave-delay --delay 1m --interval 15s --defaults-file-node2/my.sandbox.cnf 
17:03:16 slave running 0 seconds behind 

17:03:16 STOP SLAVE until 17:04:16 at master position mysql-bin.000001/2081 
17:03:31 slave stopped at master position mysql-bin.000001/2081 

17:03:46 slave stopped at master position mysql-bin.000001/2081 

17:04:01 slave stopped at master position mysql-bin.000001/2081 

17:04:16 no new binlog events 

17:04:31 slave stopped at master position mysql-bin.000001/8063 

17:04:46 slave stopped at master position mysql-bin.000001/18465 

17:05:01 slave stopped at master position mysql-bin.000001/23666 

17:05:16 slave stopped at master position mysql-bin.000001/23666 

17:05:31 START SLAVE until master 17:04:31 mysql-bin.000001/8063 








NOTE MySQL 5.6 provides delayed replication as a core feature using the 
CHANGE MASTER TO MASTER, DELAY command. See http://dev.mysql 
.com/doc/refman/5.6/en/replication-delayed.html for more details. 


References 
Documentation: http://www.percona.com/doc/percona-toolkit/pt-slave- 
delay.html. 


pt-slave-find 

This utility will connect to a MySQL master and print the replication topol- 
ogy and additional summary information. This also finds nested master/ 
slaves in a MySOL topology. For example, using the virtual environment 
from the appendix: 


$ pt-slave-find u-root,p-passwd --host-alpha 


alpha 

Version 5.6.5-m8-log 

Server ID 51 

Uptime 33:42 (started 2012-06-25T14:55:39 

Replication Is not a slave, has 1 slaves connected, is not read_only 


Filters 


Binary logging 
Slave status 
Slave mode 
Auto-increment 
InnoDB version 
beta 
Version 


+e 
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STATEMENT 


STRICT 
increment 1, 
1.2.5 


offset 1 


5.6.5-m8-log 


Server ID 52 
Uptime 14:30 (started 2012-06-25T15:14:51) 
Replication Is a slave, has 1 slaves connected, is not read_only 
Filters 
Binary logging STATEMENT 
Slave status 0 seconds behind, running, no errors 
Slave mode STRICT 
Auto-increment increment 1, offset 1 
InnoDB version 1.2.5 
+- gamma 
Version 5.6.5-m8 
Server ID 53 
Uptime 12:29 (started 2012-06-25T15:16:52) 
Replication Is a slave, has 0 slaves connected, is not read only 
Filters 


Binary logging 
Slave status 
Slave mode 
Auto-increment 
InnoDB version 


STATEMENT 

0 seconds behind, running, no errors 
STRICT 

increment 1, offset 1 

1.2.5 


When used in MySQL Sandbox, the output is not as expected for a mas- 
ter server with two connected slaves: 
$ cd SHOME/sandboxes/rsandbox 5 5 24 


$ pt-slave-find --defaults-file-master/my.sandbox.cnf --host localhost \ 
--report-format summary 


localhost 
Version 
Server ID 
Uptime 
Replication 


Filters 

Binary logging 
Slave status 
Slave mode 
Auto-increment 
InnoDB version 


5.5.24-log 

RE 

22:45 (started 16:33:12) 
Is not a slave, has 2 slaves connected, 


is not read only 
STATEMENT 
STRICT 


increment 1, 
1.1.8 


offset 1 


This was logged as a bug with Percona Toolkit; see https://bugs.launch- 
pad.net/percona-toolkit/+bug/1002512. 


CAUTION Do not always rely on the output of third-party utilities. The 
software may have bugs, be incomplete, or operate differently in a test 
environment, as demonstrated here with multiple MySQL instances on a single 
server. Adequate testing is always advisable. 
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References 
Documentation: http://www.percona.com/doc/percona-toolkit/2.1/pt-slave- 
find.html. 


Maatkit 

The predecessor to Percona Toolkit, Maatkit is no longer developed or 
maintained and has a number of commands that have not been incorpo- 
rated. It is unclear if this is due to the tools' limitations or stability. Some 
Maatkit tools are specifically deprecated by the authors as incomplete 
tools and are no longer recommended for use. 


MySQL Workbench Utilities 


The MySQL Workbench is a graphical user interface (GUI) for designing 
and managing your MySQL database objects. In addition to a fully func- 
tional entity relationship (ER) visual modeling tool, a migration tool from 
other RDBMS products, and a tool to reverse engineer MySQL schemas, 
MySQL Workbench also includes a number of command line utilities. These 
utilities are written in Python and are available under a GNU GPL v2 license. 
More information can be found at https://launchpad.net/mysql-utilities. 


NOTE MySQL version 5.6 is a Development Milestone Release (DMR). This 
clearly means this is not production-ready software, and is subject to change. 
These MySQL Workbench utilities are also works in progress. Some utilities 
are more mature than others, and in some situations the functionality is not 
complete. While it would be ideal to demonstrate the full functionality of all 
tools, the release of this book is not aligned with the unknown future release 
date of MySQL 5.6. 


CAUTION Consider the MySQL Workbench Utilities as development 
software. Every attempt has been made in this chapter to show the likely use; 
however, this is subject to change. Examples provided at the time of publication 
do show reported errors that will change in the future. 


Installation 

The MySQL Workbench Utilities are included with MySQL Workbench. 
This can be downloaded from http://www.mysql.com/downloads/work- 
bench/. In addition the MySOL Workbench Utilities are individually 
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available on Launchpad; however, there is no single download file avail- 
able at the time of this publication. This requires the use of Bazaar version 
control software to obtain these files. Refer to http://bazaar.canonical.com/ 
for download instructions for your operating system. The following com- 
mands will install these utilities on a VirtualBox virtual server, as defined in 


the appendix: 

$ cd /tmp 

$ sudo apt-get install -y bzr 

$ bzr branch lp:mysql-utilities 
$ cd mysql-utilities 

$ sudo python setup.py install 
$ mysqlrplshow --version 


MySQL Utilities mysqlrplshow version 1.0.6-preview 
Copyright (c) 2012, Oracle and/or its affiliates. All 
rights reserved. 


The following list shows all of the utilities that are currently available at 
the time of publication. In addition to the following utilities that are dis- 
cussed, additional commands support database management for importing, 
exporting, and comparison schemas and data. 


$ ls -1 /usr/local/bin/mysql* 

-rwxrwxr-x 1 uid gid 6415 Jun 14 17:29 /usr/local/bin/mysqldbcompare 
-YWXIWXYI-X uid gid 7257 Jun 14 17:29 /usr/local/bin/mysqldbcopy 

uid gid 8474 Jun 14 17:29 /usr/local/bin/mysqldbexport 
uid gid 6581 Jun 14 17:29 /usr/local/bin/mysqldbimport 
uid gid 5516 Jun 14 17:29 /usr/local/bin/mysqldiff 

uid gid 5818 Jun 14 17:29 /usr/local/bin/mysqldiskusage 
uid gid 6992 Jun 14 17:29 /usr/local/bin/mysqlfailover 
uid gid 5098 Jun 14 17:29 /usr/local/bin/mysqlindexcheck 
uid gid 3979 Jun 14 17:29 /usr/local/bin/mysqlmetagrep 
uid gid 3821 Jun 14 17:29 /usr/local/bin/mysqlprocgrep 
uid gid 5133 Jun 14 17:29 /usr/local/bin/mysqlreplicate 
uid gid 7640 Jun 14 17:29 /usr/local/bin/mysqlrpladmin 
uid gid 4294 Jun 14 17:29 /usr/local/bin/mysqlrplcheck 
uid gid 4708 Jun 14 17:29 /usr/local/bin/mysqlrplshow 
uid gid 4954 Jun 14 17:29 /usr/local/bin/mysqlserverclone 
uid gid 4119 Jun 14 17:29 /usr/local/bin/mysqlserverinfo 
uid gid 5391 Jun 14 17:29 /usr/local/bin/mysqluserclone 


-YWXIWXI-X 
-YWXIWXI-X 
-YWXIWXI-X 
-YWXIWXI-X 
-YWXIWXI-X 
-YWXIWXI-X 
-rYWXIWXI-X 
-rYWXIWXI-X 
-YWXIWXI-X 
-YWXIWXI-X 
-YWXIWXI-X 
-YWXIWXI-X 
-rYWXIWXI-X 
-YWXIWXI-X 











|)Ábhppppppmpmpmpmnpmnpmpnunm 


-YWXIWXI-X 
These utilities also require the MySOL Python/Connector to operate. 
More information can be found at https://launchpad.net/myconnpy. 


$ cd /tmp 
$ wget https://launchpad.net/myconnpy/0.3/0.3.2/+download/ 
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mysql-connector-python-0.3.2-devel.tar.gz 
$ tar xvfz mysql-connector-python-0.3.2-devel.tar.gz 
$ cd mysql-connector-python-0.3.2-devel/ 
$ sudo python setup.py install 


Documentation 
Documentation for the MySQL Utilities can be found at http://dev.mysql 
.com/doc/workbench/en/mysql-utilities.html. Additional documentation 
is included with the utilities; however, this must be generated from the 
source using Sphinx. The following steps will install Linux man pages for 
the MySQL Utilities: 


cd /tmp/mysql-utilities 

sudo apt-get install python-setuptools 

sudo easy_install -U Sphinx 

sudo python setup.py build sphinx -b man 

sudo python setup.py build man 

sudo cp build/sphinx/man/* /usr/local/man/manl 
man mysqlreplicate 


WWW Ur Ur UY 


Refer to http://sphinx.pocoo.org/ for additional instructions regarding 
installing and configuring Sphinx. 


mysqireplicate 
The following command will set up the necessary user permissions and 
settings for replication between two servers. These MySQL servers must 
have the following minimum configuration already defined in order to 
complete the replication configuration. Using the virtual environment as 
defined in the appendix: 

Server 1: 
[mysqld] 


server-id=51 
log-bin 


Server 2: 


[mysqld] 
server-id=52 


This command will perform the necessary steps to have a master/slave 
replication environment: 
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$ mysqlreplicate --master=root:passwd@alpha \ 
--slave=root:passwd@beta \ 
--rpl-user=repl:repl --pedantic 

master on alpha: ... connected. 

slave on beta: ... connected. 

Checking for binary logging on master... 

Setting up replication... 

...done. 


This can be verified with SHOW SLAVE STATUS: 


$ mysql -uroot -ppasswd -hbeta -e "SHOW SLAVE STATUS\G" 


kckckckckckckckckckckckckckckckckckckck kc ck ck k k kk 7. row KERR KKK KKK KEE KKK KKK KEK kc kc k kk 


+e tk oce HE od 


Slave_IO State: Waiting for master to send event 
Master Host: alpha 
Master User: repl 
Master Port: 3306 
Connect Retry: 60 
Master Log File: alpha-bin.000002 
Read Master Log Pos: 147 
Relay Log File: beta-relay-bin.000005 
Relay Log Pos: 349 
Relay Master Log File: alpha-bin.000002 
Slave IO Running: Yes 
Slave SOL Running: Yes 








If the necessary MySQL configuration is not in place, you will receive 
errors, including: 


$ mysqlreplicate --master=root:passwd@alpha \ 
--slave=root:passwd@beta \ 
--rpl-user-repl:repl \--pedantic 

# master on alpha: ... connected. 

# slave on beta: ... connected. 

ERROR: Slave server id is set to 0. 


This utility can be run remotely. This does not have to be executed on the 
master or slave host that is specified. The use of - -pedantic verifies the 
list of storage engines are consistent between the master and slave. You can 
also use this utility to reinitialize a MySQL slave and retrieve necessary bi- 
nary log information from a given master. This will be demonstrated later. 

A second nested master/slave relationship can be created in the same 
MySQL topology with: 
$ mysqlreplicate --master-root:passwdGbeta \ 


--Slave-root:passwdGgamma \ 
--rpl-user-repl:repl --pedantic 


120 Effective MySQL: Replication Techniques in Depth 


mysqlrplshow 
This utility shows the replication topology of your MySQL instances. Using 
the MySOL Sandbox example configuration: 


$ cd SHOME/sandboxes/rsandbox 5 5 24 

$ mysqlrplshow --master-msandbox:msandboxelocalhost:21379 

# master on localhost: ... connected. 

# Finding slaves for master: localhost:21379 

# Replication Topology Graph 

localhost:21379 (MASTER) 
| 


+--- SBslavel:21380 - (SLAVE) 


+--- SBslave2:21381 - (SLAVE) 





NOTE The output uses the - - report -host information on the slave to 
present a name. This is not a required parameter in MySQL. If not defined, the 
output will report an unknown host. 


Using the previously configured replication setup on the virtual server 
environment that was defined with the mysql replicate command: 


$ mysqlrplshow --master=root:passwd@alpha --recurse 


# Replication Topology Graph 
alpha:3306 (MASTER) 


+--- beta:3306 - (SLAVE + MASTER) 


L gamma:3306 - (SLAVE) 

This utility can be run remotely. This does not have to be executed on 
the master host that is specified. This utility is capable of traversing a full 
topology using the --recurse option as shown, providing the top level 
master is specified. All slaves must define the optional report -host with 
a value that matches the physical hostname. This option will also identify 
circular topologies. 


mysqlrplcheck 

This utility performs a sanity check on the MySQL configuration between 
a master and a slave: 

$ mysqlrplcheck --master=root:passwd@alpha --slave=root :passwd@beta 

# master on alpha: ... connected. 


# slave on beta: ... connected. 
Test Description Status 
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Checking for binary logging on master pass 
Are there binlog exceptions? pass 
Replication user exists? pass 
Checking server id values pass 
Checking server uuid values pass 
Is slave connected to master? pass 
Check master information file [WARN] 
Cannot read master information file from a remote machine. 

Checking InnoDB compatibility pass 
Checking storage engines compatibility pass 
Checking lower case table names settings pass 
Checking slave delay (seconds behind master) pass 
4 ...done. 


The warning message is because the utility has no present capability to 
connect physically to the slave server. When using master-info-rep 
-TABLE in the master and slave configuration, the following occurs. 


Is slave connected to master? [pass] 


Check master information file [pass] 
Checking InnoDB compatibility [pass] 


There is currently a reported issue regarding replication usernames and 
host wildcards, as shown when using MySQL Sandbox: 
$ cd SHOME/sandboxes/rsandbox 5 5 24 


$ mysqlrplcheck --master-msandbox:msandboxelocalhost:21379 \ 
--slave=msandbox:msandbox@localhost : 21380 


# master on localhost: ... connected. 
# slave on localhost: ... connected. 
Test Description Status 
Replication user exists? [FAIL] 


The replication user rsandbox@127.0.0.1 was not found on the master. 


Contrary to the reported FAIL message, a replication user does exist 
using a wildcard hostname within MySQL Sandbox: 


$ ./m 

master» SELECT host,user,password FROM mysql.user; 

Se ere Poss SoS sores foe SPS SSS SSS SSS Sia SSS See SS SSeS S + 
| host | user | password 

aa a PoSeeoeoe ses ss PIR SeSe SSeS SSeS SSeS Se See eee SSeS + 
| 127.% | rsandbox | *B07EB15A2E7BD9620DAE47B194D5B9DBA14377AD | 
pecumemteete gesmeeceeemeO duerme eem iiem Eee mE * 


This example shows both the strengths and weaknesses of any open 
source utilities. If there was an error in MySOL replication, the running of 
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this command regularly (e.g., daily) is a good management monitoring 
approach. However, when the tool reports a problem that is not actually a 
problem (due to a software bug), this can complicate the benefits of a 
monitoring approach. 


CAUTION This is an example where software may have bugs, be incomplete, 
or operate differently in a test environment as demonstrated here. Adequate 
testing is always advisable. 


mysqirpladmin 
The mysqlrpladmin utility provides a number of commands for managing 
a MySQL topology. These commands currently include 


e start Start replication on all slaves specified. 
e stop Stop replication on all slaves specified. 


e reset Stop and reset replication on all slaves specified. 


health Display the replication health of the defined master and 
slave topology. 


gtid Verify the status of global transaction identifier (GTID) 
variables to ensure these are correctly configured for the defined 
masters and slaves. This command also displays UUID information 
for all specified servers. 


elect Perform a best slave election and report which slave to use for 
switchover. 


e switchover Perform a slave promotion of an elected slave to master 
and reconfigure the existing master as a slave. 


e failover Conduct a failover from an unavailable master to the best 
available slave. 


Prerequisite Configuration 
When using the elect, switchover, or failover commands you will have to en- 
able the following settings in all of the servers’ my . cnf files to support GTID: 


[mysqid] 

log-bin 

log-slave-updates 
disable-gtid-unsafe-statements 
gtid-mode-ON 
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If you fail to include the log-slave-updates option on the master, or 
log-bin and log-slave-updates on slaves, the following error will be 
found in the MySQL error log: 

120614 18:51:42 [ERROR] --gtid-mode-ON or UPGRADE STEP 1 or UPGRADE STEP 2 
requires --log-bin and --log-slave-updates 

Refer to Chapter 3 for more information about the GTID configuration 
settings. 

If GTID is not correctly configured on all hosts, you will run into the fol- 
lowing replication error when running SHOW SLAVE STATUS\G: 


Slave» SHOW SLAVE STATUS\G 


Last IO Errno: 1593 
Last_IO Error: The slave IO thread stops because the master has 
GTID_MODE OFF and this server has GTID_MODE ON 


health command 
The following example shows the health of a master/slave configuration: 
mysqlrpladmin --master=root:passwd@alpha --slave=root:passwd@beta health 


$ 
# Checking privileges. 
# Replication Topology Health: 





pese Paradan demere muon ecce em Jee euim + 
| host port | role | state | gtid_mode | health 
.-------- Ed Poses SSS 4-------- dciechlcdeec Poses esses + 
| alpha 3306 | MASTER | UP | OFF | OK | 
| beta 3306 | SLAVE | UP | OFF | OK | 
deve panen dp d$--—————— dle pce xem * 
# done 


NOTE You must specify all the slave servers you wish to check. This utility does 
not discover what slaves exist in the replication topology. 


If there are any issues with the slaves operating correctly you will find 
output similar to the following: 


$ cd $HOME/sandboxes/rsandbox_5 5 24 
$ mysqlrpladmin --master=msandbox:msandbox@localhost:21379 \ 
--slaves=msandbox:msandbox@localhost:21380, \ 
msandbox:msandbox@localhost:21381 health 
# Checking privileges. 
# Replication Topology Health: 


+------------ +------- +--------- +------- +------------ +-------------------------- + 
| host | port | role | state | gtid_mode | health | 
+------------ +------- +--------- +------- +------------ +-------------------------- + 
| localhost | 21379 | MASTER | UP | No | OK | 
| localhost | 21380 | SLAVE | WARN | Cannot connect to slave. 
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| localhost | 21381 | SLAVE WARN | Cannot connect to slave. 


When using the health command for this utility, this can be run remotely. 

The following example shows two slaves connected in a MySQL envi- 
ronment using a GTID setup. This example shows an error situation when 
one slave is not correctly configured: 


$ mysqlrpladmin --master=root:passwd@alpha \ 

--slave=root :passwd@beta, root:passwd@gamma health 
# Checking privileges. 
# Replication Topology Health: 





oof SSeS SSeS Pee Ve SS SS ESS SS SSS Loo SSeS SSS SS LSS SSS SSS SSS SS + 
| host | port | role | state | gtid_mode | health 
Sha aia! Poco e Ss Pore RS pore {prin Sie SI Se is Pore arses + 
| alpha | 3306 | MASTER | UP | ON | OK | 
| beta | 3306 | SLAVE | UP | ON | OK | 
gamma | 3306 | SLAVE | UP | OFF | Not connected | 
duc nre = niei ae SS Se Se SSS SaaS aS SSS SS SS ta = ee ee a + 
# done 


NOTE For the global transaction identifier (GTID) to work correctly in a 
MySQL topology all servers must be configured accordingly. 


Alternatively, if GTID is correctly configured but one or more of the slave 
threads is not running, the following will be reported. With this grid display 
output it is important to review all columns for different error conditions: 


# Checking privileges. 





# 
# Replication Topology Health: 
+-------- +------- +--------- +-------- +------------ +--------- + 
| host | port | role | state | gtid mode | health | 
+-------- +------- +--------- +-------- +------------ +--------- + 

alpha | 3306 | MASTER | UP | ON | OK 
| beta | 3306 | SLAVE | UP | ON | OK | 
| gamma | 3306 | SLAVE | UP | ON | ERROR | 
+-------- +------- +--------- +-------- +------------ +--------- + 

The utility will also provide feedback on any slave lag. For example: 
+------- +------ +-------- +------- +------ +--------------------------------------- 
| host | port | role | state | gtid | health 
+------- +------ +-------- +------- +------ +--------------------------------------- 
| alpha | 3306 | MASTER | UP | ON | OK 
| beta | 3306 | SLAVE | UP | ON | Slave has 1 transactions behind master 
| gamma | 3306 | SLAVE | WARN | | Cannot connect to slave. 
+------- +------ +-------- +------- +------ +--------------------------------------- 
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gtid Command 

The enabling of GTID with gtid_mode=ON, log-slave-updates and 
disable-gtid-unsafe-statements configuration settings will sup- 
port the most impressive feature of these utilities, that is, the ability for a 
master switchover or failover to a slave. You can confirm the GTID configu- 
ration for the full MySOL topology is valid with: 


alpha$ mysqlrpladmin --master=root:passwd@alpha --slave=root:passwd@beta gtid 


# 
# UUIDS for all servers: 


desees en iocos re cen ed pcne um E + 
| host | port | role | uuid 

paanan ma Pots Ses dp—m—————— a + 

| alpha | 3306 | MASTER | ba7ac732-b707-11e1-a1b3-0800275824dc 

| beta | 3306 | SLAVE | 0941e912-b709-11e1-albc-0800278bd7a3 | 
4-------- 4------- pue ER queue HE EIE CIUS + 

# 

# Transactions executed on the server: 

jese—e- oie tS, g-————e--- oS SoS = SoS SSS m See SSS See SSS See SSS SS SSS + 
| host | port | role | gtid | 
.-------- 4------- pocesesens Sac ae al ae + 
| alpha | 3306 | MASTER | BA7AC732-B707-11E1-A1B3-0800275824DC:1 | 
| beta | 3306 | SLAVE | 0941E912-B709-11E1-A1BC-0800278BD7A3:1-4 | 
| beta | 3306 | SLAVE | BA7AC732-B707-11E1-A1B3-0800275824DC:1 | 
4-------- 4------- 4--------- p c aaa aaa + 


Refer to Chapter 3 for the details of what MySQL configuration vari- 
ables are necessary for correct GTID configuration and usage. If GTID has 
not been configured correctly in your MySQL topology, the following types 
of error messages can be presented: 


alphas mysqlrpladmin --master=root:passwd@alpha \ 
--slave=root:passwd@beta gtid 
# Checking privileges. 

# WARNING: GTIDs are not supported on this topology. 
# ...done. 


alphas mysqlrpladmin --master=root:passwd@alpha V 
--slave=root :passwd@beta, root:passwde@gamma gtid 





# Checking privileges. 
# UUIDS for all servers: 


dme E dmm POSH SSS SSeS SS Se ecu SS See See ee eS + 
| host | port | role | uuid 

eene eme panee Sie pene (oS SSeS SSeS eS Se SSeS See Se SS SSeS + 
| alpha | 3306 | MASTER | ba7ac732-b707-11e1-a1b3-0800275824dc 

| beta | 3306 | SLAVE | 0941e912-b709-11e1-albc-0800278bd7a3 

| gamma | 3306 | SLAVE | e6193aa0-b714-11e1-a209-080027530628 | 


# ERROR retrieving GTID information: Global Transaction IDs are not enabled. 


# ...done. 
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This command will not operate correctly when run remotely. The follow- 


ing information was obtained when the command was run on the master 


(i.e., alpha) server. 


reset Command 


This command will reset replication for the specified instance. Following 
the correction of the required MySQL configuration on the second slave to 
support GTID, and restarting the MySQL instance, we can use a combina- 
tion of these utilities described to correctly configure and confirm replica- 


tion in our MySQL topology. For example, to reset replication on a MySQL 


slave: 


+HtHHHH MH 


mysqlrpladmin --slave=root:passwd@gamma reset 
Checking privileges. 
Performing STOP on all slaves. 

Executing stop on slave gamma:3306 Ok 
Performing RESET on all slaves. 

Executing reset on slave gamma:3306 Ok 
...done. 


To reconfigure the MySQL server as a slave with the MySQL topology: 


$ mysqlreplicate --master=root:passwd@alpha \ 
--slave=root :passwd@gamma \ 
--rpl-user=repl:repl --pedantic 

# master on alpha: ... connected. 

# slave on gamma: ... connected. 

# Checking for binary logging on master... 

# Setting up replication... 

# ...done. 

To confirm that the slave is now correctly configured and operating in 
the MySQL topology: 
$ mysqlrpladmin --master=root :passwd@alpha \ 





--sSlave=root :passwd@beta, root :passwd@gamma health 
Checking privileges. 


Replication Topology Health: 





EHE qpeseeeedqpuesseeeeemepeueaesenigesememememepememeeemeg 
host | port | role | state | gtid mode | health | 
TERHERE pese eee SS SSS SS SSeS SSS SS SS eae ES EE 
alpha | 3306 | MASTER | UP | ON | OK 
beta | 3306 | SLAVE | UP | ON | OK 
gamma | 3306 | SLAVE | UP | ON | OK 
SSS PaaS SSS Sa Se eee a aS eee Se eS Se ee jore SS Sa ee 
done 
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TIP These new MySQL 5.6 Utilities can help in providing easy command line 
configuration and management of your MySQL topology, eliminating the 
legacy approach that required more administration management. It is 
recommended that you understand how to leverage mysqladmin, and the 
various commands that provide replication-specific information, with the 
mysql client to learn how these tools actually work for verification. 


elect Command 

This command will provide information about which slave is the best to 
promote to a master. In order to support switchover and failover with these 
utilities the following MySQL 5.6 prerequisites are necessary. You should 
also ensure that your hardware environment is the same for high volume 
environments to ensure a slave can support future master load. 


e GTID is correctly configured and operating with the master and all 
slaves in the MySOL topology. 


e All slaves are correctly configured as crash-safe slaves. 


You can confirm how a switchover would operate by verifying what 
slave would be elected: 


$ mysqlrpladmin -vv --master-root:passwdealpha \ 
--slave=root:passwd@beta,root:passwd@gamma elect 
Checking privileges. 
Electing candidate slave from known slaves. 
Checking eligibility of slave beta:3306 for candidate. 
Slave connected to master ... Ok 
GTID MODE-ON ... Ok 
Logging filters agree ... Ok 
Replication user exists ... Ok 
Best slave found is located on beta:3306. 


te th cb cb cb db dod 


te 


...done. 


switchover Command 

When the MySQL topology can determine a new slave can be elected, it is 
now possible to perform a controlled switchover from the master to a dif- 
ferent slave in the MySQL replication topology. This will also reconfigure 
the current master as a slave. 


$ mysqlrpladmin -vv --master=root:passwd@alpha \ 
--new-master=root:passwd@beta --demote-master switchover 
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Checking privileges. 
Performing switchover from master at alpha:3306 to slave at beta:3306. 
Checking candidate slave prerequisites. 
GTID MODE-ON is set for all servers. 
Checking eligibility of slave beta:3306 for candidate. 
Slave connected to master ... Ok 
GTID MODE-ON ... Ok 
Logging filters agree ... Ok 
Replication user exists ... Ok 


Creating replication user if it does not exist. 

Blocking writes on master. 

LOCK STRING: FLUSH TABLES WITH READ LOCK 

Waiting for slaves to catch up to old master. 

Stopping slaves. 

Performing STOP on all slaves. 

UNLOCK STRING: UNLOCK TABLES 

Demoting old master to be a slave to the new master. 
Switching slaves to new master. 

Executing CHANGE MASTER on alpha:3306. 

CHANGE MASTER TO MASTER HOST - 'beta', MASTER USER - 'repl', 
MASTER PASSWORD - 'repl', MASTER PORT - 3306, MASTER AUTO POSITION-1 


Se dt db db db db db db db dt db db db db dt dt dt dt dt dt 


Starting all slaves. 
Performing START on all slaves. 
Checking slaves for errors. 


# 

# 

# 

# Switchover complete. 

# Attempting to contact beta ... Success 
# 
# 


Replication Topology Health: 


gaea 4------ donec 4------- 4----- 4-------- preesse 4------ 
| host | port | role | state | gtid| health | version | 
L------ oes dee————— gue dme deeem doeet ce k ae e euer eos 
| beta | 3306 | MASTER | UP | ON | OK | 5.6.5-m8-log | beta- 
| alpha| 3306 | SLAVE | UP | ON | OK | 5.6.5-m8-log | alpha 
4------ 4------ T-------- 4------- 4----- 4-------- ducc EE 4------ 
# done 


You should verify the health of the new replication topology appropri- 
ately defining the new master and slave hosts. 


$ mysglrpladmin --master=root:passwd@beta V 
--slave=root :passwd@alpha, root:passwd@gamma health 


# Checking privileges. 

# 

# Replication Topology Health: 

+-------- +------- +--------- +-------- dperureieuoi peres ps PrSesseeeesessseeeeesese 
| host | port | role | state | gtid mode | health 

deem 4 pe mmm dmm demum Rm m dimmi i mmm em 
| beta | 3306 | MASTER | UP | ON | OK 

| alpha | 3306 | SLAVE | UP | on | OK 

| gamma | 3306 | SLAVE | WARN | | Slave is not connected 
pose dees 4e eet À dem denim mtm mm decimum in tim intimi im i 
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As you can see, in this switchover example, the second slave was not cor- 
rectly assigned to the new master. Under normal execution this would 
occur. This is demonstrated to show an error message. 


failover Command 
Finally, the failover command can be used when the master is no longer 
accessible and a suitable slave is elected to become the master: 


$ mysglrpladmin -vv --slave=root:passwd@beta,root:passwd@gamma failover 
# Checking privileges. 

# Performing failover. 

# Checking eligibility of slave beta:3306 for candidate. 

# — GTID MODE-ON ... Ok 

# Replication user exists ... Ok 

# Candidate slave beta:3306 will become the new master. 

# Preparing candidate for failover. 

# LOCK STRING: FLUSH TABLES WITH READ LOCK 

# Connecting candidate to gamma:3306 as a master to retrieve unprocessed GTIDs. 
# Change master command for beta:3306 

# CHANGE MASTER TO MASTER HOST = 'gamma', MASTER USER = 'repl', 

MASTER PASSWORD - 'repl', MASTER PORT - 3306, MASTER AUTO POSITION-1 


4 UNLOCK STRING: UNLOCK TABLES 

4 Waiting for candidate to catch up to slave gamma:3306. 
# Slave beta:3306: 

# QUERY = SELECT SQL THREAD WAIT AFTER GTIDS( 
'0C50EA14-E74E-11E1-9C7E-0800275824DC:1-5', 3) 

# Return Code = 0 
# Slave beta:3306: 
# QUERY = SELECT SQL THREAD WAIT AFTER GTIDS( 
'11603F93-E74E-11E1-9C7E-0800273BF04E:1-6', 3) 








Return Code - None 
Creating replication user if it does not exist. 





Stopping slaves. 





Performing STOP on all slaves. 


Executing stop on slave gamma:3306 Ok 

Switching slaves to new master. 

Change master command for beta:3306 

CHANGE MASTER TO MASTER HOST - 'beta', MASTER USER - 'repl', 
MASTER PASSWORD - 'repl', MASTER PORT - 3306, MASTER AUTO POSITION-1 
4 Change master command for gamma:3306 
4 CHANGE MASTER TO MASTER HOST - 'beta', MASTER USER - 'repl', 
MASTER PASSWORD - 'repl', MASTER PORT - 3306, MASTER AUTO POSITION-1 


# 
# 
# 
# 
# Executing stop on slave beta:3306 Ok 
# 
# 
# 
# 


# Starting slaves. 

# Performing START on all slaves. 
# Executing start on slave gamma:3306 Ok 
# Checking slaves for errors. 
# gamma:3306 status: Ok 
# Failover complete. 

# Attempting to contact beta ... Success 

# Attempting to contact gamma ... Success 
# 
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mysqlfailover 

This script provides continuous replication monitoring and enables auto- 
matic failover to a designated or most up-to-date slave in the event of an 
unplanned outage on the master for MySQL 5.6. Promotion policies are 
configurable, but the default policy is to promote the most current slave in 
the topology based on the global transaction identifier (GTID) that was 
discussed in detail in Chapter 3. Think of mysqlfailover as a daemon 
that will automatically fail over your environment in the event of a master 
failure. 

The mysqlfailover utility will report as follows in this case: 


$ mysqlfailover --master=root :passwd@alpha:3306 \ 
--slaves=root :passwd@beta: 3306, root: passwd@gamma:3306 \ 
--failover-mode=auto 
MySQL Replication Failover Utility 
Failover Mode = auto Next Interval = Wed Aug 15 23:50:44 2012 
Master Information 
Binary Log File Position Binlog Do DB Binlog Ignore DB 
alpha-bin.000004 187 
Replication Health Status 





hose = aS SoS s SERE SS Se Looe SSeS SSeS dccem * 
| host | port | role | state | gtid mode | health | 
PRO aes Poor aes pope Sea iS porn eee PRIS Ree SRS Pir Seer + 
| alpha | 3306 | MASTER | UP | ON | OK | 
| beta | 3306 | SLAVE | UP | ON | OK | 
| gamma | 3306 | SLAVE | UP | ON | OK | 
poee se ses pes SS S25 qpesemcmss- Se PosSsese sess = $SSs Seese Ss + 


Q-quit R-refresh H-health G-GTID Lists U-UUIDs 


When using the mysql failover utility a new table is added only to the 
master host in the mysql schema called failover console. This table 
contains information about the master host: 


master» SHOW CREATE TABLE mysql.failover_console\G 
Ckckckckckckckckckckckckckckckckck ck k kk kkk |l. row XXX kk kk kk ck ck kk kk kc kc kckckokok 
Table: failover console 
Create Table: CREATE TABLE ^failover console" ( 

^host^ char(30) DEFAULT NULL, 

^port^ char(10) DEFAULT NULL 
) ENGINE-InnoDB DEFAULT CHARSET-utf8 























master» SELECT * FROM mysql.failover_console\G 
Ckckckckckckckckckckckckckckck ck ck ck k kk kkk |. row KERR kk kkck kk ck kk kk kk kk kk 
host: alpha 

port: 3306 
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Simulating Master Failure 

When the mysqld process and the mysqld safe angel process are killed 
on the master, the mysql failover script will perform the failover to a suit- 
able slave within a few seconds. The following information is shown during 
this step: 


Failover starting in 'auto' mode... 

Candidate slave beta:3306 will become the new master. 
Preparing candidate for failover. 

Creating replication user if it does not exist. 
Stopping slaves. 

Performing STOP on all slaves. 

Switching slaves to new master. 

Starting slaves. 

Performing START on all slaves. 

Checking slaves for errors. 

Failover complete. 

Failover console will restart in 5 seconds. 





dE cb db dt dt db db db dto dt 





MySQL Replication Failover Utility 

Failover Mode - auto Next Interval - Thu Aug 16 00:20:18 2012 
Master Information 

Binary Log File Position Binlog Do DB Binlog Ignore DB 
beta-bin.000003 6985 

Transactions executed on the servers: 


deter peeeneuem qae eS sss pase sess Poses sss SeS peser t 
| host | port | role | state | gtid mode | health | 
dose posing POSSE Sees S A + 
| beta | 3306 | MASTER | UP | ON | OK | 
| gamma | 3306 | SLAVE | UP | ON | OK | 
poe eset LL LLL Pore see Honts Hnn prera ben nnn mmm + 


You can then reset and include the original master in the topology with 
the following commands: 


CAUTION The following commands show the syntax to reset a MySQL 
instance as a slave. This is for demonstration purposes. In a production 
situation, you will need to implement an appropriate recovery of your dataset 


from a backup. 

$ mysqlrpladmin --slave=root:passwd@alpha reset 

$ mysqlreplicate --master=root:passwd@beta \ 
--slave=root:passwd@alpha --rpl-user-repl:repl --pedantic 

# master on beta: ... connected. 

# slave on alpha: ... connected. 

# Checking for binary logging on master... 

# Setting up replication... 

# ...done. 

$ mysqlrpladmin --master=root:passwd@beta \ 


--slave=root :passwd@alpha, root:passwd@gamma health 
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# Checking privileges. 

# 

# Replication Topology Health: 

queens +------- PSS eee Ens 4-------- doe uc hee Sis RRES + 
| host | port | role | state | gtid_mode | health 
Peso +s SRS Pron RSE SS pinssi $RS RSPR SSE PHP See RRR SE + 
| beta | 3306 | MASTER | UP | ON | OK | 
| alpha | 3306 | SLAVE | UP | ON | OK | 
| gamma | 3306 | SLAVE | UP | ON | OK | 
+-------- +------- +--------- +-------- +------------ +----------- + 
# done 


Replication Failover Managers 


MySQL replication easily enables a read scalable architecture by support- 
ing multiple slaves. This provides a straightforward approach for replacing 
a failed MySQL slave for high availability. It does not, by default, provide 
high availability for the primary master database that performs all writes. 
It is possible to define a MySQL replication configuration where failover, 
and possibly failback, is possible, thus enabling high availability on the 
MySQL master. 

Prior to MySQL 5.6, with the introduction of new features including 
GTID and crash-safe slaves, a more complicated approach was necessary. 
In a controlled situation it is possible with a correct configuration to manu- 
ally perform a failover with these steps: 


e Ensure appropriate MySQL configuration to support failover and 
failback operations. 


e Move slaves from the primary master to the failover master with 
appropriate replication checks. 


Stop write access to the primary master and set the MySQL instance 
to read only on the running server and in the server configuration file. 


Confirm the failover master has received and performed all 
replication events. 


Disable read-only access on the MySQL failover master. 


Update the default startup state of the original MySQL master and 
failover master. 
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e Redirect application and batch write operations to the failover 
master. There is no single way to manage this step. This depends on 
the application technology in use. A general approach is via a virtual 
IP (VIP). 


The setup of a MySQL pair and the regular use of performing a con- 
trolled failover and failback is an excellent way to be prepared for what 
steps are necessary in a true failover situation for your disaster recovery 
(DR) strategy. This is a popular approach for managing software upgrades, 
database modifications, and application upgrades. The configuration of a 
MySQL pair is discussed in detail in Chapter 4 on multi-master replica- 
tion. In a failure situation, however, there are several edge cases that can 
cause potential data loss and corruption in different disaster scenarios. A 
number of tools are available in the MySQL ecosystem to help in the man- 
agement and automation of failover. 


MySQL MHA 


Created by MySQL expert and Oracle ACE Director Yoshinori Matsunobu 
(http://yoshinorimatsunobu.blogspot.com/), the MySQL-MHA: MySQL 
Master High Availability manager and tools (MySQL MHA) provide a 
wrapper to support the management of a MySQL replication and failover 
environment. This is one of the newer tools available and is being actively 
developed and used. This tool is written in Perl. 

A primary objective of MHA is automating master failover and slave 
promotion within short (usually 10 to 30 seconds) downtime, without suf- 
fering from replication consistency problems, without spending money for 
lots of new servers, without performance penalty, without complexity (easy 
to install), and without changing existing deployments. 


Node Software Installation 

On each MySQL master and slave server and management server, the in- 
stallation of the MHA node software is necessary. You should always check 
for the most current version to download at http://code.google.com/p/ 
mysql-master-ha/. 
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Ubuntu/Debian 


$ sudo apt-get install libdbd-mysql-perl 
$ cd /tmp 
$ wget http://mysql-master-ha.googlecode.com/files/ 
mha4mysql-node 0.53 all.deb 

sudo dpkg -i mha4mysql-node *.deb 

rm -f i mha4mysql-node *.deb 





Ur ur 


RHEL/CentOS/OL 


$ sudo yum install perl-DBD-MySQL 

$ cd /tmp 

$ wget https://mysql-master-ha.googlecode.com/files/ 
mha4mysql-node-0.53-0.noarch.rpm 

sudo rpm -ivh mha4mysql-node-*.noarch.rpm 

rm -f i mha4mysql-node_*.noarch.rpm 


$ 
$ 


Source Code 
Refer to http://code.google.com/p/mysql-master-ha/wiki/Installation for 
instructions on installing MySQL MHA from the source code. 


Manager Software Installation 

MHA is a management process and should be operated on a separate 
management server that is not part of the MySQL master/slave topology. 
The installation of the MHA manager software is performed with the 
following. 


Ubuntu/Debian 


$ sudo apt-get install -y libdbd-mysql-perl libconfig-tiny-perl \ 
liblog-dispatch-perl libparallel-forkmanager-perl 

cd /tmp 

wget http://mysql-master-ha.googlecode.com/files/ 
mha4mysql-manager 0.53 all.deb 

sudo dpkg -i mha4mysql-manager * all.deb 

rm -f mha4mysql-manager_* all.deb 


$ 
$ 
$ 
$ 


RHEL/CentOS/OL 


$ sudo yum install perl-Config-Tinyl perl-Log-Dispatch perl-Parallel-ForkManager 
$ cd /tmp 
$ wget https://mysql-master-ha.googlecode.com/files/ 
mha4mysql -manager-0.53-0.noarch.rpm 
$ sudo rpm -ivh mha4mysql-manager-*.noarch.rpm 
$ rm -f mha4mysql-manager-*.noarch.rpm 
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References 
MHA is an open source project available under the GPL v2 license. More 
information can be found at the following links: 


e Code http://code.google.com/p/mysgl-master-ha/ 


e Blog http://yoshinorimatsunobu.blogspot.com/2011/07/ 
announcing-mysql-mha-mysql-master-high.html 


Usage with Virtual Environment 

In this example we will use the three virtual servers that are defined in the 
instructions in the appendix. The alpha server is the MySQL master, the 
beta server is the failover master, and the gamma server will be used for a 
management server. In this example we will use MySQL version 5.5. 


Configuration Setup MHA requires a number of configuration settings 
regarding the definition of the MySQL topology and the SSH and MySQL 
communication between the servers. On the management server the 
following is required: 


$ cd SHOME 

$ mkdir -p mha/etc 

$ echo "[server default] 

# mysql user and password 

user-root 

password-passwd 

ssh user-user 

# working directory on the manager 
manager workdir-/home/user/mha/log 

# working directory on MySQL servers 
remote workdir-/home/user/mha/log 
master binlog dir-/home/user/mysql/data 
[server1] 

hostname-alpha 

[server2] 

hostname-beta" > SHOME/mha/etc/mha.cnf 
$ sudo touch /etc/masterha default.cnf 


SSH Access 
MHA uses SSH for server communication between databases. This will 
require you to set up SSH keyed authentication to enable automated 
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commands. The following MHA command will check and confirm access 
between the management server and all MySOL nodes: 


$ masterha check ssh --conf-$HOME/mha/etc/mha.cnf 

info] Reading default configurations from /etc/masterha default.cnf.. 
info] Reading application default configurations from 
/home/user/mha/etc/mha.cnf.. 

info] Reading server configurations from /home/user/mha/etc/mha.cnf.. 
info] Starting SSH connection tests.. 

debug] Connecting via SSH from userGalpha(192.168.1.51:22) to 
user@beta(192.168.1.52:22).. 

debug] ok. 

debug] Connecting via SSH from user@beta(192.168.1.52:22) to 
user@alpha(192.168.1.51:22).. 

debug] ok. 

info] All SSH connection tests passed successfully. 





MySQL Configuration Optimizations 
The default slave configurations as defined in the appendix will produce 
the following warning when running masterha_check_repl1: 
[warning] relay log purge=0 is not set on slave 
beta(192.168.1.52:3306). 
This can be corrected with the following additional MySOL configura- 
tion and restarting the MySQL instances on all servers: 
# SHOME/mysql/etc/my.cnf 
[mysqld] 
relay log purge=0 
The manager requires mysqlbinlog in the PATH on each server to run 
successfully via SSH. While the mysqldump command may be in the PATH 
on the management server, and also on the MySQL instances when you 
connect manually, you may experience the following error: 
[info] Connecting to user@192.168.1.52(beta:22).. 
Can't exec "mysqlbinlog": No such file or directory at 
/usr/share/perl5/MHA/BinlogManager.pm line 99. 


mysqlbinlog version not found! 
at /usr/bin/apply diff relay logs line 463 





The SSH command used here is a non-interactive shell, and while the 
/etc/profile.d/mysql.sh script is executed for interactive shell us- 
age, it is not for non-interactive usage (i.e., if you manually ssh to a server, 
the MYSOL HOME environment variable is defined and $MYSOL 
HOMEBE/bin is added to your PATH). 
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The solution is to add to the start of the SHOME/ . bashrc file a reference 
to source this file specifically: 


[ -s /etc/profile.d/mysql.sh ] && . /etc/profile.d/mysql.sh 


Adding to the end of the file will not work with the virtual environments 
configured, as Ubuntu uses this line in the user $HOME/ . bashrc file to 
stop reading the file in non-interactive mode: 

# If not running interactively, don't do anything 
[ -z "SPS1" ] && return 

By default design, the slave replication stream on the active master serv- 
er is not running. The masterha check repl will correctly report this 
with: 


[warning] SQL Thread is stopped(no error) on alpha(192.168.1.51:3306) 


This can be solved by starting the slave replication stream on the active 
master: 


alpha» SLAVE START; 


MHA Replication Check 
When the MHA node and manager software are installed and configured 
you can confirm your MySQL topology meets the needs of MHA with: 


$ masterha check repl --conf-$HOME/mha/etc/mha.cnf 

info] Reading default configurations from /etc/masterha default.cnf.. 
info] Reading application default configurations from 
/home/user/mha/etc/mha.cnf.. 

info] Reading server configurations from /home/user/mha/etc/mha.cnf.. 
info] MHA::MasterMonitor version 0.53. 

info] Multi-master configuration is detected. Current primary (writable) 
master is alpha(192.168.1.51:3306) 

info] Master configurations are as below: 

Master alpha(192.168.1.51:3306), replicating from beta(192.168.1.52:3306) 
Master beta(192.168.1.52:3306), replicating from alpha(192.168.1.51:3306), 
read-only 

info] Dead Servers: 

info] Alive Servers: 


info alpha(192.168.1.51:3306) 

info beta(192.168.1.52:3306) 

info] Alive Slaves: 

info beta(192.168.1.52:3306) Version=5.5.24-log (oldest major version 


between slaves) log-bin:enabled 

info Replicating from alpha(192.168.1.51:3306) 
info] Current Alive Master: alpha(192.168.1.51:3306) 
info] Checking slave configurations.. 








info] Checking replication filtering settings.. 
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info Version check ok. 


info binlog do db- , binlog ignore db- 

info Replication filtering check ok. 

info] Starting SSH connection tests.. 

info] All SSH connection tests passed successfully. 
info] Checking MHA Node version.. 


info] Checking SSH publickey authentication settings on the current master.. 
info] HealthCheck: SSH to alpha is reachable. 

info] Master MHA Node version is 0.53. 

info] Checking recovery script configurations on the current master.. 

info Executing command: save binary logs --command-test --start pos-4 


--binlog dir-/home/user/mysql/data 


--output file-/home/user/mha/log/save binary logs test 
--manager version-0.53 --start file-mysql-bin.000005 








info Connecting to user@alpha(alpha).. 
Creating /home/user/mha/log if not exists.. ok. 
Checking output directory is accessible or not.. 


ok. 


Binlog found at /home/user/mysql/data, up to mysql-bin.000005 


[info] Master setting check done. 


[info] Checking SSH publickey authentication and checking recovery 


script configurations on all alive slave servers.. 





[info Executing command apply diff relay logs --command-test 

--slave user-root --slave host-beta --slave ip-192.168.1.52 
--slave_port=3306 --workdir=/home/user/mha/log --target_version=5.5.24-log 
--manager_version=0.53 --relay_log_info=/home/user/mysql/data/relay-log.info 
--relay dir-/home/user/mysql/data/  --slave pass-xxx 


[info] Connecting to user8192.168.1.52(beta:22).. 
Creating directory /home/user/mha/log.. done. 
Checking slave recovery environment settings.. 


Opening /home/user/mysql/data/relay-log.info ... ok. 
Relay log found at /home/user/mysql/data, up to relay-log.000012 
Temporary relay log file is /home/user/mysql/data/relay-log.000012 


Testing mysql connection and privileges.. done. 
Testing mysqlbinlog output.. done. 
Cleaning up test file(s).. done. 
info] Slaves settings check done. 
info 
alpha (current master) 
*--beta 
info] Checking replication health on beta.. 
info ok. 
warning] master ip failover script is not defined. 
warning] shutdown script is not defined. 








info] Got exit code 0 (Not master dead). 
MySQL Replication Health is OK. 

$ echo $? 

0 


If MySQL is not running you will find errors similar to: 


[error] [/usr/share/perl5/MHA/ServerManager.pm, 1n188] There is no alive 


We can't do failover 


server. 
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[error] [/usr/share/per15/MHA/MasterMonitor.pm, 1n383] Error happened on checking 
configurations. at /usr/share/perl5/MHA/MasterMonitor.pm line 298 

[error] [/usr/share/per15/MHA/MasterMonitor.pm, 1n478] Error happened on 
monitoring servers. 

[info] Got exit code 1 (Not master dead). 

MySQL Replication Health is NOT OK! 

$ echo $? 

i 


Running the MHA Manager 

After successfully configuring SSH and MySQL access for MHA and con- 
firming the replication environment is suitable to be managed by MHA, 
you can start the MHA manager with: 


$ masterha manager --conf=$HOME/mha/etc/mha.cnf 


info] Slaves settings check done. 

info] 

alpha (current master) 

*--beta 

warning] master ip failover script is not defined. 

warning] shutdown script is not defined. 

info] Set master ping interval 3 seconds. 

warning] secondary check script is not defined. It is highly recommended 
setting it to check master reachability from two or more routes. 
info] Starting ping health check on alpha(192.168.1.51:3306).. 
info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond.. 





This command is run in the foreground in this situation and will not 
return access to the session until completed. For production operation, this 
command should be wrapped to create an applicable daemon process and 
monitored accordingly. The warnings described will be discussed at a later 
time. 

In a new terminal session you can verify the operation of the manager 
that was just started. This command should be added to applicable system 
and MySQL monitoring to detect any issues or failure with the manager. 
$ masterha check status --conf-$HOME/mha/etc/mha.cnf 
mha (pid:16387) is running(0:PING OK), master:alpha 


$ echo $? 
0 


If the MHA manager is not running, the following will occur: 


$ masterha check status --conf-$HOME/mha/etc/mha.cnf 
mha is stopped(2:NOT RUNNING). 

$ echo $? 

2 
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Stopping the MHA Manager 
The running MHA manager can be terminated with: 


$ masterha stop --conf-$HOME/mha/etc/mha.cnf 
Stopped mha successfully. 


MySOL MHA Operation 


The power of using MySOL MHA is to handle the situation of an uncon- 
trolled failover (i.e., the loss of the MySQL master). MHA can also be used 
to perform a controlled failover. This can be very helpful in managing soft- 
ware releases and database maintenance windows. 


MHA Controlled Failover 
You can simulate a controlled failover with the following command on the 
management server: 


$ masterha master switch --conf-$HOME/mha/etc/mha.cnf V 

--master state-alive --new master host-beta --orig master is new slave \ 

--interactive=0 

info] MHA::MasterRotate version 0.53. 
info] Starting online master switch.. 
info] * Phase 1: Configuration Check Phase.. 
info] Reading default configurations from /etc/masterha_default.cnf.. 
info] Reading application default configurations from 
/home/user/mha/etc/mha.cnf.. 
info] Reading server configurations from /home/user/mha/etc/mha.cnf.. 
info] Multi-master configuration is detected. Current primary (writable) 
master is alpha(192.168.1.51:3306) 
info] Master configurations are as below: 
Master alpha(192.168.1.51:3306), replicating from beta(192.168.1.52:3306) 
Master beta(192.168.1.52:3306), replicating from alpha(192.168.1.51:3306), 
read-only 
info] Current Alive Master: alpha(192.168.1.51:3306) 
info] Alive Slaves: 





info beta(192.168.1.52:3306) Version=5.5.24-log (oldest major version 
between slaves) log-bin:enabled 
info Replicating from alpha(192.168.1.51:3306) 


info] Executing FLUSH NO WRITE TO BINLOG TABLES. This may take long time.. 
info ok. 

info] Checking MHA is not monitoring or doing failover.. 

info] Checking replication health on beta.. 

info ok. 

info] beta can be new master. 








info 
From: 
alpha (current master) 
+--beta 


Tos 
beta 
*--a 
info 
info 
info 
info 
info 
info 


info 


info 
info 
info 
info 
info 
beta( 
info 
info 
info 
info 


info 
info 
info 
info 
info 
info 
info 
info 


info 
info 
info 
info 
info 





info 
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(new master) 

pha 
Checking whether beta(192.168.1.52:3306) is ok for the new master.. 
ok. 
** phase 1: Configuration Check Phase completed. 


* Phase 2: Rejecting updates Phase.. 


warning] master ip online change script is not defined. 





Skipping disabling writes on the current master. 


Locking all tables on the orig master to reject updates 


from everybody (including root): 


Executing FLUSH TABLES WITH READ LOCK.. 
ok. 
Orig master binlog:pos is mysql-bin.000002:107. 
Waiting to execute all relay logs on beta(192.168.1.52:3306).. 
master pos wait (mysgl-bin.000002:107) completed on 
92.168.1.52:3306). Executed 0 events. 
done. 
Getting new master's binlog name and position.. 
mysql-bin.000002:107 
All other slaves should start replication from here. 


Statement should be: CHANGE MASTER TO MASTER HOST-'beta or 192.168.1.52', 
MASTER PORT-3306, MASTER LOG FILE-'mysql-bin.000002', MASTER LOG POS-107, 
MASTER USER-'repl', MASTER PASSWORD-'xxx'; 


Setting read only-0 on beta(192.168.1.52:3306).. 
ok. 
* Switching slaves in parallel.. 
Unlocking all tables on the orig master: 
Executing UNLOCK TABLES.. 
ok. 
Starting orig master as a new slave.. 
Resetting slave alpha(192.168.1.51:3306) and starting replication 


from the new master beta(192.168.1.52:3306).. 


Executed CHANGE MASTER. 
Slave started. 
All new slave servers switched successfully. 
* Phase 5: New master cleanup phase.. 
beta: Resetting slave info succeeded. 
Switching master to beta(192.168.1.52:3306) completed successfully. 





This command also has an interactive mode. This can be used with 


--interactive-1. 


MHA Automated Failover 
In the defined virtual environment from the appendix that is being used, 


we can test MHA by stopping MySQL on the master: 


alpha$ mysqladmin shutdown 
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The following output from the MySOL MHA Manager session occurs 
within a few seconds: 


[warning] Got error on MySQL select ping: 2006 (MySQL server has gone away) 
[info] Executing SSH check script: save binary logs --command-test --start pos- 
4 --binlog dir=/home/user/mysql/data --output file-/home/user/mha/10g/ 
Save binary logs test --manager version-0.53 --binlog prefix=mysql-bin 
Creating /home/user/mha/log if not exists.. ok. 
Checking output directory is accessible or not.. 
ok. 
Binlog found at /home/user/mysql/data, up to mysql-bin.000005 
info] HealthCheck: SSH to alpha is reachable. 
warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on 
'192.168.1.51' (111)) 
warning] Connection failed 1 time(s).. 
warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on 
'192.168.1.51" (111)) 
warning] Connection failed 2 time(s).. 
warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on 
'192.168.1.51' (111)) 
warning] Connection failed 3 time(s).. 





warning] Master is not reachable from health checker! 





warning] Master alpha(192.168.1.51:3306) is not reachable! 








warning] SSH is reachable. 

info] Connecting to a master server failed. Reading configuration file /etc/ 
masterha default.cnf and /home/user/mha/etc/mha.cnf again, and trying to connect 
to all servers to check server status.. 

info] Reading default configurations from /etc/masterha default.cnf.. 

info] Reading application default configurations from /home/user/mha/etc/mha. 
ent... 
info] Reading server configurations from /home/user/mha/etc/mha.cnf.. 
info] Dead Servers: 


info alpha (192.168.1.51:3306) 

info] Alive Servers: 

info beta (192.168.1.52:3306) 

info] Alive Slaves: 

info beta(192.168.1.52:3306) Version=5.5.24-log (oldest major version 
between slaves) log-bin:enabled 

info Replicating from alpha(192.168.1.51:3306) 


info] Checking slave configurations.. 

info] Checking replication filtering settings.. 

info Replication filtering check ok. 

info] Master is down! 

info] Terminating monitoring script. 

info] Got exit code 20 (Master dead). 

info] Reading default configurations from /etc/masterha_default.cnf.. 

info] Reading application default configurations from /home/user/mha/etc/mha. 
enf.. 
info] Reading server configurations from /home/user/mha/etc/mha.cnf.. 
info] MHA::MasterFailover version 0.53. 

info] Starting master failover. 

info 








info] * Phase 1: Configuration Check Phase.. 
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info 
info] Dead Servers: 

info alpha(192.168.1.51:3306) 

info] Checking master reachability via mysql (double check).. 
info ok. 

info] Alive Servers: 


info beta(192.168.1.52:3306) 

info] Alive Slaves: 

info beta(192.168.1.52:3306) Version=5.5.24-log (oldest major version 
between slaves) log-bin:enabled 

info Replicating from alpha(192.168.1.51:3306) 

info] ** Phase 1: Configuration Check Phase completed. 

info 

info] * Phase 2: Dead Master Shutdown Phase.. 

info 





info] Forcing shutdown so that applications never connect to the current master 
warning] master ip failover script is not set. Skipping invalidating dead 
master ip address. 

warning] shutdown script is not set. Skipping explicit shutting down of the 
dead master. 

info] * Phase 2: Dead Master Shutdown Phase completed. 

info 
info] * Phase 3: Master Recovery Phase.. 
info 
info] * Phase 3.1: Getting Latest Slaves Phase.. 
info 
info] The latest binary log file/position on all slaves is mysql-bin.000005:107 
info] Latest slaves (Slaves that received relay log files to the latest): 


info beta(192.168.1.52:3306) Version=5.5.24-log (oldest major version 
between slaves) log-bin:enabled 
info Replicating from alpha(192.168.1.51:3306) 


info] The oldest binary log file/position on all slaves is mysql-bin.000005:107 
info] Oldest slaves: 


info beta(192.168.1.52:3306) Version=5.5.24-log (oldest major version 
between slaves) log-bin:enabled 

info Replicating from alpha(192.168.1.51:3306) 

info 


info] * Phase 3.2: Saving Dead Master's Binlog Phase.. 
info 
info] Fetching dead master's binary logs.. 





info] Executing command on the dead master alpha(192.168.1.51:3306): save - 
binary logs --command-save --start file-mysqgl-bin.000005 --start_pos=107 





--binlog dir-z/home/user/mysql/data --output file-/home/user/mha/log/ 
Saved master binlog from alpha 3306 20120617202902.binlog --handle raw binlog-1 
--disable log bin-0 --manager version-0.53 





Creating /home/user/mha/log if not exists.. ok. 

Concat binary/relay logs from mysql-bin.000005 pos 107 to mysql-bin.000005 EOF 
into /home/user/mha/log/saved master binlog from alpha 3306 20120617202902. 
binlog 





Dumping binlog format description event, from position 0 to 107.. ok. 
Dumping effective binlog data from /home/user/mysql/data/mysql-bin.000005 
position 107 to tail(126).. ok. 
Concat succeeded. 
saved master binlog from alpha 3306 20120617202902.binlog 
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100$ 126 0.1KB/s 00:00 

info] scp from user9192.168.1.51:/home/user/mha/log/saved master binlog from 
alpha 3306 Á 

20120617202902.binlog to local:/home/user/mha/log/saved master binlog from 
alpha 3306 20120617202902.binlog succeeded. 

info] HealthCheck: SSH to beta is reachable. 

info 








info] * Phase 3.3: Determining New Master Phase.. 
info 
info] Finding the latest slave that has all relay logs for recovering other 
slaves.. 

info] All slaves received relay logs to the same position. No need to resync 
each other. 

info] Searching new master from slaves.. 

info Candidate masters from the configuration file: 

info Non-candidate masters: 

info] New master is beta(192.168.1.52:3306) 

info] Starting master failover.. 








info 
From: 
alpha (current master) 
+--beta 


To: 
beta (new master) 
info 
info] * Phase 3.3: New Master Diff Log Generation Phase.. 
info 
info This server has all relay logs. No need to generate diff files from 
the latest slave. 

info] Sending binlog.. 

saved master binlog from alpha 3306 20120617202902.binlog 

100$ 126 0.1KB/s 00:00 

info] scp from local:/home/user/mha/log/saved master binlog from al- 

pha 3306 20120617202902. 

binlog to userGbeta:/home/user/mha/log/saved master binlog from alpha 

3306 20120617202902.binlog succeeded. 

info 











info] * Phase 3.4: Master Log Apply Phase.. 
info 
info] *NOTICE: If any error happens from this phase, manual recovery is needed. 
info] Starting recovery on beta(192.168.1.52:3306).. 

info Generating diffs succeeded. 

info] Waiting until all relay logs are applied. 

info done. 

info] Getting slave status.. 

info] This slave(beta)'s Exec Master Log Pos equals to Read Master Log Pos( 
mysql-bin.000005:107). No need to recover from Exec Master Log Pos. 

info] Connecting to the target slave host beta, running recover script.. 








info] Executing command: apply diff relay logs --command-apply --slave user- 
root --slave host-beta --slave ip-192.168.1.52 --slave_port=3306 --apply files 
-/home/user/mha/log/saved master binlog from alpha 3306 20120617202902.binlog 
--workdir-/home/user/mha/log --target_version=5.5.24-log 
--timestamp-20120617202902 --handle raw binlog-1 --disable log bin-0 
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--manager version-0.53 --slave pass-xxx 

[info] 

Applying differential binary/relay log files /home/user/mha/log/saved master 
binlog from alpha 3306 20120617202902.binlog 

on beta:3306. This may take long time... 

Applying log files succeeded. 

[info] All relay logs were successfully applied. 

[info] Getting new master's binlog name and position.. 

[info] mysql-bin.000003:107 

[info] All other slaves should start replication from here. Statement should 
be: CHANGE MASTER TO MASTER HOST-'beta or 192.168.1.52', MASTER PORT-3306, 
MASTER LOG FILE-'mysql-bin.000003', MASTER LOG POS-107, MASTER USER-'repl', 
MASTER PASSWORD-'xxx'; 


warning] master ip failover script is not set. Skipping taking over new 
master ip address. 

info] Setting read only-0 on beta(192.168.1.52:3306).. 

info ok. 

info] ** Finished master recovery successfully. 

info] * Phase 3: Master Recovery Phase completed. 

info 
info] * Phase 4: Slaves Recovery Phase.. 
info 
info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase.. 
info 
info] Generating relay diff files from the latest slave succeeded. 
info 





info] * Phase 4.2: Starting Parallel Slave Log Apply Phase.. 
info 
info] All new slave servers recovered successfully. 
info 
info] * Phase 5: New master cleanup phase.. 
info 
info] Resetting slave info on the new master.. 

info beta: Resetting slave info succeeded. 

info] Master failover to beta(192.168.1.52:3306) completed successfully. 
info 








----- Failover Report ----- 

mha: MySQL Master failover alpha to beta succeeded 

Master alpha is down! 

Check MHA Manager logs at gamma for details. 

Started automated(non-interactive) failover. 

The latest slave beta(192.168.1.52:3306) has all relay logs for recovery. 
Selected beta as a new master. 

beta: OK: Applying all logs succeeded. 

Generating relay diff files from the latest slave succeeded. 

beta: Resetting slave info succeeded. 

Master failover to beta(192.168.1.52:3306) completed successfully. 


As you can see in this first simple example, MySOL MHA successfully 
detected a loss of communication with the master, and after performing 


necessary checks and validations has successfully failed over to the 
slave server. 
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Additional MHA Configuration Settings 

There are several additional configuration options and scripts that can fur- 
ther improve the high availability with MHA. These include the master 
ip failover script, shutdown script, and secondary check 
script options, which are indicated as warnings in the previous output. 


e master ip failover script In common high availability (HA) 
environments, in many cases people allocate one virtual IP address 
on a master. If the master crashes, HA software moves the virtual IP 
address to the standby server. 


A sample script is located under (MHA Manager package)/samples/ 
scripts/master ip failover. Sample scripts are included in the MHA Man- 
ager tarball and GitHub branch. 

MHA Manager calls master ip failover script three times. The 
first time is before entering the master monitor (for script validity check- 
ing), the second time is just before calling shutdown script, and the 
third time is after applying all relay logs to the new master. 


e shutdown script You may want to force a shutdown of the master 
MySQL instance so that it never restarts the instance. This is known 
as node fencing. This is important to avoid split-brain. A sample 
script is located at (MHA Manager package)/samples/scripts/ 
power manager. Sample scripts are included in MHA Manager 
tarball and GitHub branch. 


e secondary check script In general, it is highly recommended to have 
two or more network routes to check MySQL master server availability. 
By default, MHA Manager checks only using a single route from the 
manager to the master. This is not recommended in a highly available 
production environment. MHA can actually have two or more routes 
for checking network connectivity by calling an external script defined 
with the secondary check script parameter. For example: 


Secondary check script - masterha secondary check -s remote 1 -s remote 2 


MMM 


The Multi-Master Replication Manager for MySQL (MMM) is a set of 
scripts that provide management monitoring and failover of a MySOL rep- 
lication environment. 


MySQL Replication Tools 147 


While this product works in a number of situations, generally just two serv- 
ers, and is still used with production systems, it is no longer maintained and 
several important bugs remain unresolved. It is not recommended you use 
this product. For a good explanation of current issues, see http://mysql-mmm 
.org/mmm1:development-plans and http://www.xaprb.com/blog/2011/05/04/ 
whats-wrong-with-mmm/. 

More information about MMM can be found at http://mysql-mmm.org/. 


Flipper 


Another historical product found in MySQL environments for managing 
MySQL pairs is Flipper. This can still be found in production use. This 
product is also no longer maintained, and online documentation from the 
original creator, Proven Scaling, is no longer available. You can find the 
source code at http://code.google.com/p/flipper/ and more information 
regarding usage at http://ronaldbradford.com/blog/using-flipper-to- 
manage-mysg|l-pairs-2008-12-09/. 


Cluster Control 


The team from Severalnines provides a wizard approach to create an appro- 
priate MySQL configuration for a highly available MySQL replication envi- 
ronment. This requires the use of the online web form to obtain details of 
your current configuration to create a suitable configuration, which you can 
download. No testing of this wizard has been performed by the author. For 
more details, visit http://www.severalnines.com/replication-configurator/. 


Cluster Management 


A common approach for using a MySQL pair effectively in a production 
application environment is to use virtual IPs (VIPs) management for dedi- 
cated write and read threads. By using a VIP, application connections can 
remain defined as a fixed value, while the cluster management determines 
the host ownership of the VIP address. 

For a database like MySQL, connection management is more complex 
due to several factors, including transactions and persistent connections. An 
incorrect use of VIP can cause problems like split-brain situations and data 


148 Effective MySQL: Replication Techniques in Depth 


corruption. The use of a VIP can be problematic, with persistent connection 
managers and hanging connections. Depending on the application technol- 
ogy, additional connection management, including killing the connection 
pool on application servers, may be necessary to minimize risks. This follow- 
ing article gives some great background details related to MySQL: http:// 
scale-out-blog.blogspot.com/2011/01/virtual-ip-addresses-and-their.html. 

The Linux High Availability project maintains the building blocks for 
cluster infrastructure, including the Heartbeat daemon and Cluster Glue. 
More information can be found at http://linux-ha.org/. Combined with 
Pacemaker, this Cluster Resource Manager (CRM) helps to provide a suit- 
able solution to be used in conjunction with MySQL replication. More 
information can be found at http://clusterlabs.org. 

Red Hat also provides a high availability cluster manager. See http:// 
www.redhat.com/products/enterprise-linux-add-ons/high-availability/ 
for more information. 

There is some information about clustering MySQL instances using Or- 
acle Clusterware. More details can be found at http://ilmarkerm.blogspot 
.com/2011/11/clustering-mysql-instances-with-oracle.html. 

Providing a highly availability cluster environment combined with a 
MySQL replication topology is further complicated when the application 
technology uses persistent connection management (e.g., Java). Any clus- 
ter management implementation needs to consider the type of database 
access and the applicable error detection of a transaction and applicable 
recovery processes to retry the transaction or report an appropriate error. 
With persistent connection management technology it is often of benefit to 
flush the connection pool when changing the underlying host. 


Percona Replication Manager (PRM) 


By combining Corosync, Pacemaker, a MySQL resource agent and MySQL, 
the Percona Replication Manager (PRM) provides a cluster management 
solution with a distributed architecture to reduce a single point of failure 
(SPOF). The goal of PRM is to provide the following features: 


e Reader and writer VIP behaviors similar to MMM. 


e If the master fails, a new master is promoted from the slaves; no 
master to master setup needed. Selection of master is based on 
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scores published by the slaves, the more up to date slaves have 
higher scores for promotion. 


Some nodes can be dedicated to be only slaves or less likely to 
become master. 


€ A node can be the preferred master. 


If replication on a slave breaks or lags beyond a defined threshold, 
the reader VIP(s) is removed. MySQL is not restarted. 


If no slaves are OK, all VIPs, readers and writer, will be located on 
the master. 


During a master switch, connections are killed on the demoted 
master to avoid replication conflicts. 


All slaves are in read-only mode. 


Simple administrative commands can remove master role from a node. 


Pacemaker stonith devices are supported. 
e No logical limits in term of number of nodes. 
e Easy to add nodes. 


An introduction presentation at  http://www.percona.com/files/ 
presentations/percona-live/dc-2012/PLDC2012-high-availability-with- 
percona-prm.pdf provides an overview of the PRM product and imple- 
mentation approach. Detailed instructions for configuration and usage is 
available from the documentation at https://github.com/jayjanssen/Perco- 
na-Pacemaker-Resource-Agents/blob/master/doc/PRM-setup-guide.rst. 
The outdated article at http://www.mysqlperformanceblog.com/2011/ 
11/29/percona-replication-manager-a-solution-for-mysql-high-availability- 
with-replication-using-pacemaker/ provided the listed features shown 
and a very detailed description of how PRM is designed to work. 


Replication Prefetch 


Paul Tuckfield of YouTube presented to the MySQL Community in 2007 the 
concept of replication slave prefetch. A very simple idea, this improves the 
performance of the single-threaded slave SQL thread by preloading 
the necessary data pages by a separate read thread. The ability to predict 
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the future in MySQL replication is possible because the master binary log 


and the relay log contain all events yet to be applied in the slave replication 


stream. The following is a code snippet of the work: 


module(..., package.seeall); required "lusql.mysql" 

pattern- { 
["UPDATESS-($w-2).**$S(WHERE.*)"] = "SELECT * FROM $1 $22", 
[DELETESS-FROM$S-($w-).$S(WHERE.*)"] = SELECT * FROM $1 22", 


) 


env 
con 


lusql.mysql.() 
env:connect("test", "root", "*****", "localhost", mysql.port) 


"oil 


function before write(event) 
local line - event.query 
if not line then return end 
for pat,repl in pairs(pattern) do 


local str - string.gsub(line, pat, repl) 
if str then con:execute(str); break; end 


end 


end 


You can preview this code on a page from MySQL High Availability 
(O'Reilly, 2011) at Safari Books online at http://j.mp/EM3-prefetch (no sub- 
scription required). 


Several articles exist that can provide some input on the various options 


that have been implemented by different individuals or companies: 


e Replication Booster by Yoshinori Matsunobu can be found at https:// 


github.com/yoshinorim/replication-booster-for-mysql. Some more 
information is at http://yoshinorimatsunobu.blogspot.com/2011/10/ 
making-slave-pre-fetching-work-better.html. 


Domas Mituzas writes about work developments at Facebook at 
http://dom.as/2011/12/03/replication-prefetching/. The code is 
available at https://launchpad.net/mysglatfacebook/tools. 


e Anders Karlsson has created a MySOL replication accelerator. Read 


more at http://karlssonondatabases.blogspot.com/2011/03/want-to- 
accellerate-mysql-slave-here-is.html. The code is available at http:// 
sourceforge.net/projects/slavereadahead/. 


It is not recommended that mk-slave-prefetch be used. This is 
deprecated as directed by the author of the utility. More information 
can be found at http://bit.ly/MOraRd. 
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MySQL Patches and Variants 


Patches for MySQL are driven by the necessity of solving a specific prob- 
lem that the core product may or may not include in the future. These 
improvements are generally due to the long deployment time of including 
a feature in the core MySOL product that serves for all general use cases, 
not just the specific customer case that it was originally developed for. 

Patches are provided for the benefit of others, which is great for the 
community; however, these come at a cost if the needs or specific versions 
do not match your environment. When these conditions do match, the ad- 
vantages of the development can be of great benefit. For example, Domas 
Mituzas wrote about how convenient it was to use the Google Patch for 
MySQL 4.0.26 because Wikipedia used the exact same point release. Read 
more at http://dom.as/2007/06/23/mysql-40-google-edition/ and http:// 
dom.as/2010/07/25/mysql-versions-at-wikipedia/. 

As you can see by these following company names, large web presences 
use and extend MySQL for their respective businesses and for the possible 
benefit of other MySQL users. 


Independent Community Users 


Several large MySQL users have contributed greatly to the MySQL ecosys- 
tem. Some of the patches provided have become core features in more 
recent versions of MySQL. Led initially by Google as early as 2007, many 
community provided patches have included improvements around replica- 
tion. The Google patches for MySQL version 5.0 included global transaction 
IDs, binary logging event checksums, and semisynchronous replication, 
now standard features in MySQL 5.6. Facebook is the top organization cur- 
rently providing MySQL patches to the MySQL community. With several of 
the top experts in MySQL and InnoDB internals outside of Oracle, Facebook 
has a continued investment in improving MySQL at scale, specifically in 
InnoDB multicore usage and replication. 

While these patches can provide some exciting benefits, they have been 
specifically written for the needs of the client in question. They may not be 
applicable to other workloads and environments, and without a team of 
specialists that know the internal working code of MySQL may be a high 
risk to implement. 
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References 
The following are some examples of community contributed works: 


e Google http://code.google.com/p/google-mysgl-tools/wiki/ 
Mysql5Patches 


e Facebook Mhttp://www.facebook.com/MySOLatFacebook and 
https://launchpad.net/mysqlatfacebook 


e eBay http://code.google.com/p/mysql-heap-dynamic-rows/ 
e Twitter https://github.com/twitter/mysql 


Commercial Organizations 


The MySQL ecosystem has seen several companies start to provide addi- 
tional engineering services for supporting new features with MySQL. 
These provide feature development for a fee, and generally provide a far 
quicker time to market. 


Monty Program 

Founded by Michael "Monty" Widenius, one of the original founders of 
MySQL, Monty Program retains a number of core engineers from the origi- 
nal MySQL AB company. They provide many improvements to the MySQL 
product and offer these patches for upstream integration in future official 
MySQL versions. MariaDB is a feature compatible drop-in replacement ver- 
sion of MySQL that also includes many additional storage engines. MariaDB 
is available under the open source GNU GPL v2 license. More information 
can be found at http://mariadb.org. For environments that are heavy users of 
MyISAM, MariaDB provides a number of great performance improvements. 


Percona 

Percona is the longest established alternative commercial vendor in the 
MySQL space. They develop and support several MySOL Products, includ- 
ing Percona Server, Percona Cluster (with Galera), XtraDB, XtraBackup, 
and Percona Toolkit. More information can be found at http://percona.com. 
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Conclusion 


A database administrator should be aware of and use this diverse set of 
tools to monitor, manage, verify, and improve an existing production 
MySQL replication environment. Combined with deployment automation 
and service monitoring as described in Chapter 8, replication can provide 
powerful benefits in any production environment when configured and 
operating normally. 

Examples and links in this chapter are available for download from 
http://EffectiveMySQL.com/book/replication-techniques. 
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Extending Replication 
for Practical Needs 


MySQL replication is essential for any production environment. In the 
previous chapters we have discussed the use of replication, some of the 
known limitations, and hinted at some of the ongoing improvements. In 
many deployments, current MySQL replication provides an adequate 
approach to improving scalability and availability. Prior to the MySOL 5.6 
version there were more complex operations necessary for multi-master 
replication and high availability management. In addition, some of the fea- 
tures of MySOL replication that are strengths in the flexibility of the repli- 
cation implementation are also limitations for certain architectures. 
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The MySQL replication landscape has significantly changed with addi- 
tional third-party providers. In this chapter we will discuss some of these 
providers and the features they offer, specifically: 


e Supporting synchronous replication 


e Active/active multi-master replication supporting bidirectional 
replication 


e Complex topologies, including circular, star, and fan, in support of 
multiple masters for a given slave 


Highly Requested Replication Features 


Without a doubt, the ability for synchronous replication ranks as one of the 
top requests for improvements with native MySQL replication functional- 
ity. The asynchronous nature provides several benefits, especially in WAN 
or network limiting environments; however, this also causes a lack of guar- 
anteed data consistency between all servers, and depending on the needs 
of the application, that can introduce additional complexity. Many scale- 
out environments use native replication successfully to support hundreds 
and thousands of MySQL instances. New features from other providers 
can add value to these existing environments. 

Providing a mixed topology that includes active master/master replication 
and supporting multiple masters for a given slave is also a common need for 
ensuring high availability and reducing latency in a global deployment. 

Another key feature request is that of automatic sharding and partitioning, 
especially for cloud-based deployments. 

Combining all these features with the need for adequate disaster recov- 
ery management and ongoing growth in data storage, throughput, and 
performance are the key feature areas for using MySQL at scale. 


MySQL Cluster 


MySQL Cluster, which is a different product than the MySQL database 
server, provides several of these features natively. Synchronous replication 
and automatic partitioning are standard features in MySQL Cluster; how- 
ever, this is a very different product than the MySQL server. The name is 
appropriate for features, yet deceiving when compared with MySQL. 
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While this product includes a SOL interface, data access for high through- 
put environments can be achieved from an Application Programming In- 
terface (API). The strengths and features of MySOL Cluster are not always 
applicable to an Online Transaction Processing (OLTP) application using 
traditional MySOL replication. There are also additional limitations, in- 
cluding the database size with available memory. This product is ideal in 
certain situations and is widely used in the telecommunications and gam- 
ing industries. The use, understanding, and benefits of MySOL Cluster are 
topics for an entire book. 

It should be noted that MySOL Cluster mainly differs from a regular 
MySQL server for the following additional reasons: 


e There is currently no referential integrity. Foreign keys are being 
developed for a future version. 


e You can only use the NDB storage engine and are then limited to 
READ-COMMITTED transaction level. 


e While primary key lookups are fast and concurrent access is faster 
than a regular MySQL server, more complex queries, including table 
joins and group by operations, are more expensive. 


For more information about MySQL Cluster, refer to http://www.mysql 
.com/products/cluster/. 


Galera Cluster for MySOL 


Galera Cluster for MySOL provides synchronous replication and multi- 
master features when using InnoDB. This means a Galera Cluster supports 
reads and writes to any node, and provides a true multi-master experience 
with no lag and no loss of transactions. 

Galera Cluster is built on a more generic replication API called wsrep 
(Write Set REPlication). The wsrep API defines an interface between the 
database server and the replication plugin and is a separate open source 
project by Codership. MySOL-wsrep is a patch for MySOL to implement 
the wsrep API in the database server. This patch will enable the MySOL 
server to use a wsrep provider plugin, for example, Galera. Galera is the 
wsrep provider including synchronous multi-master replication. 
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Galera 2.x is the currently supported version available with MySOL 5.1 
GA and MySQL 5.5 GA. Galera is an open source project available under 
the GPL v3 license, with integration components available under the GPL 
v2 license to be compatible with MySOL. 


Current Limitations 


When comparing Galera with a traditional MySQL server, there are some 
limitations; however, these are not serious roadblocks to trying and using 
Galera effectively: 


e Only InnoDB tables are supported in replication. 


e LOCK and UNLOCK statements, GET LOCK() and RELEASE LOCK() 
functions are not supported. 


e Log output to TABLE using 1og output is not supported. Logging 
must be to FILE. 


e XA (eXtended Architecture) transactions are not currently supported. 


References 
For more information see: 


e Product Home  http://www.codership.com/content/using-galera- 
cluster/ 


e Downloads http://www.codership.com/downloads/download- 
mysqlgalera 
e Documentation http://www.codership.com/wiki 


e Details of wsrep can be found at https://launchpad.net/wsrep 


Terminology 
When using Galera Cluster, a number of terms are used and referenced in 
online documentation. These include the following: 
e Galera Cluster This refers to the operational nodes that constitute 
the cluster. 
e Node This refers to an individual server instance that is part of a 


Galera Cluster. 


e Joiner This refers to a new node that wishes to join a Galera Cluster. 
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e Donor This refers to a Galera node that is used for synchronizing 
the Joiner. 


e Group This is a layer of communication between an individual 
node and a Galera Cluster. 


Installation 


Galera provides RedHat/CentOS/Oracle Linux . rpm packaging, Ubuntu/ 
Debian .deb packaging, and Linux binary .tar.gz installation options. 
The following installation steps will install the binaries on an Ubuntu/ 
Debian Linux system that is defined in the virtual environments from the 
appendix. 

The following steps use specific Galera and MySQL binary versions. 
Please refer to the downloads link for the most current versions available. 
For this installation we will be using three servers: 


e alpha 192.168.1.51 
€e beta 192.168.1.52 
e gamma 192.168.1.53 


Install Galera 
The wsrep implementation (i.e Galera) must be installed first. This is 
demonstrated on the alpha server: 


cd /tmp 

wget https://launchpad.net/galera/2.x/23.2.1/«download/galera-23.2.1-amd64.deb 
sudo apt-get install libss10.9.8 

sudo dpkg -i galera-*.deb 


Vr Xr Xr oro Ur 


rm -f galera*.deb 


The following library is installed with the Galera package. This specific 
file path is required later during configuration: 
$ ls -1 /usr/lib/galera/ 


total 2104 
-rwxr-xr-xXx 1 root root 2153064 May 18 18:02 libgalera smm.so 


The Galera package contains the following files for reference: 


$ dpkg --contents galera-23.2.1-amd64.deb 
drwxr-xr-x root/root 0 2012-05-18 18:02 ./ 
drwxr-xr-x root/root 0 2012-05-18 18:02 ./etc/ 
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drwxr-xr-x root/root 0 2012-05-18 18:02 ./etc/init.d/ 

-rwxr-xr-x root/root 3132 2012-05-18 18:02 ./etc/init.d/garb 

drwxr-xr-x root/root 0 2012-05-18 18:02 ./etc/default/ 

-rw-r--r-- root/root 502 2012-05-18 18:02 ./etc/default/garb 

drwxr-xr-x root/root 0 2012-05-18 18:02 ./etc/ld.so.conf.d/ 

-rw-r--r-- root/root 0 2012-05-18 18:02 ./etc/ld.so.conf.d/galera.conf 
drwxr-xr-x root/root 0 2012-05-18 18:02 ./usr/ 

drwxr-xr-x root/root 0 2012-05-18 18:02 ./usr/share/ 

drwxr-xr-x root/root 0 2012-05-18 18:02 ./usr/share/doc/ 

drwxr-xr-x root/root 0 2012-05-18 18:02 ./usr/share/doc/galera/ 
-rw-r--r-- root/root 35147 2012-05-18 18:02 ./usr/share/doc/galera/COPYING 
-rw-r--r-- root/root 5900 2012-05-18 18:02 ./usr/share/doc/galera/ 
README-MySQL 

-rw-r--r-- root/root 21740 2012-05-18 18:02 ./usr/share/doc/galera/README 
drwxr-xr-x root/root 0 2012-05-18 18:02 ./usr/lib/ 

drwxr-xr-x root/root 0 2012-05-18 18:02 ./usr/lib/galera/ 

-rwxr-xr-x root/root 2153064 2012-05-18 18:02 ./usr/lib/galera/ 

libgalera smm.so 

drwxr-xr-x root/root 0 2012-05-18 18:02 ./usr/bin/ 

-rwxr-xr-x root/root 1345448 2012-05-18 18:02 ./usr/bin/garbd 











Install MySQL 5.5 with wsrep Patch 
The patched MySQL version with the wsrep connector from Codership 
can now be installed with the following commands: 


cd 

sudo apt-get install libaiol # MySQL 5.5 dependency 

wget https://launchpad.net/codership-mysql/5.5/5.5.23-23.6/«download/mysq1l- 
.5.23 wsrep 23.6-linux-x86 64.tar.gz 

tar xvfz mysql-5.5.23 wsrep 23.6-linux-x86 64.tar.gz 

mv mysql-5.5.23 wsrep 23.6-linux-x86 64 mysql 


Vr or Vr UP XS oir Ur 


cd mysqi 


The installation and verification of MySOL is the same as normal 
MySQL procedures that are also described in the appendix: 


$ ./scripts/mysgl install db 

nstalling MySQL system tables... 

120606 15:02:01 [Note] WSREP: Read nil XID from storage engines, 
Skipping position init 
120606 15:02:01 [Note] WSREP: wsrep load(): loading provider library 'none' 
120606 15:02:05 [Note] WSREP: Service disconnected. 

120606 15:02:06 [Note] WSREP: Some threads may fail to exit. 

OK 
Filling help tables... 
120606 15:02:06 [Note] WSREP: Read nil XID from storage engines, 
skipping position init 
120606 15:02:06 [Note] WSREP: wsrep load(): loading provider library 'none' 
120606 15:02:06 [Note] WSREP: Service disconnected. 

120606 15:02:07 [Note] WSREP: Some threads may fail to exit. 

OK 
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$ ./bin/mysqld safe & 
$ tail -20 data/^hostname' .err 


$ ./bin/mysql secure installation 


$ HOSTNAME- hostname" 
$ echo "[client] 
user-root 
password-passwd 


[mysql] 

prompt-'$[HOSTNAME]» '" > SHOME/.my.cnf 
$ ./bin/mysql -e "SELECT version()"; 
+----------- + 

| version() | 

+----------- + 

| 5.5.23 | 

+----------- + 
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We can confirm the wsrep deployment by reviewing the new default 


variables available with: 


$ bin/mysql -e "SHOW GLOBAL VARIABLES LIKE 'ws%'" 


wsrep OSU method 

wsrep auto increment control 
wsrep causal reads 

wsrep certify nonPK 

wsrep cluster address 

wsrep cluster name 

wsrep convert LOCK to trx 
wsrep data home dir 
wsrep dbug option 

wsrep debug 

wsrep drupal 282555 workaround 
wsrep forced binlog format 
wsrep max ws rows 
wsrep max ws size 
wsrep node address 

wsrep node incoming address 
wsrep node name 

wsrep notify cmd 

wsrep on 

wsrep provider 

wsrep provider options 
wsrep recover 

wsrep replicate myisam 
wsrep retry autocommit 
wsrep slave threads 
wsrep sst auth 
wsrep sst donor 
wsrep sst method 

wsrep sst receive address 





wsrep start position 





my | 
OFF 
/home /user/mysq1/data/ 


wsrep cluster 


OFF 
OFF 
NONE 
131072 
1073741824 





AUTO 
alpha 


OFF 
none 


mysqldump 
AUTO 


00000000-0000-0000-0000-000000000000: 
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Configuring MySOL with wsrep 

MySQL will operate as a normal traditional server by default. Galera usage 
requires a number of MySQL configuration settings. The following are the 
minimum recommended settings: 


$ cd SHOME/mysql 

$ mkdir etc 

$ echo "[mysqld] 

wsrep provider-/usr/lib/galera/libgalera smm.so 
binlog format-ROW 

default storage engine-InnoDB 

innodb autoinc lock mode-2 

innodb locks unsafe for binlog-1 





# optional 
innodb flush log at trx commit-0 





innodb_doublewrite=0" > etc/my.cnf 


As you can see, there is no replication-specific configuration necessary. 
Binary logging is not required, nor is a server ID to use within a Galera 
Cluster. To define a Galera Cluster you must specify a cluster address. 
Restarting the MySQL instance with the new configuration and specifying 
a new empty cluster address with the wsrep_cluster_address configu- 
ration parameter enables Galera for this instance: 


$ ./bin/mysqladmin -uroot shutdown 
$ ./bin/mysqgld safe --defaults-file-etc/my.cnf --wsrep cluster address-gcomm:// & 


As you can see from the MySQL error log output, a lot of additional 
information is now provided: 


mysqld safe Starting mysqld daemon with databases from /home/user/mysql/data 
nnoDB: The InnoDB memory heap is disabled 

nnoDB: Mutexes and rw locks use GCC atomic builtins 

nnoDB: Compressed tables use zlib 1.2.3 

nnoDB: Using Linux native AIO 

nnoDB: Initializing buffer pool, size - 128.0M 

nnoDB: Completed initialization of buffer pool 

nnoDB: highest supported file format is Barracuda. 

nnoDB: Waiting for the background threads to start 

nnoDB: 1.1.8 started; log sequence number 1595843 

Note] Event Scheduler: Loaded 0 events 

Note] WSREP: Read nil XID from storage engines, skipping position init 
Note] WSREP: wsrep load(): loading provider library 
'/usr/lib/galera/libgalera smm.so' 

Note] WSREP: wsrep load(): Galera 23.2.1(r129) by Codership Oy 
«infoGcodership.com» loaded successfully. 

Warning] WSREP: Could not open saved state file for reading: /home/user/mysql/ 
data//grastate.dat 
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[Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1 

[Note] WSREP: Preallocating 134219048/134219048 bytes in '/home/user/mysql/ 
data//galera.cache'... 

[Note] WSREP: Passing config to GCS: base host - 192.168.1.51; 

gcache.dir = /home/user/mysql/data/; gcache.keep pages size = 0; 

gcache.mem size = 0; gcache.name = /home/user/mysql/data//galera.cache; 
gcache.page size = 128M; gcache.size = 128M; gcs.fc debug = 0; 

gcs.fc factor = 0.5; gcs.fc limit = 16; gcs.fc master slave = NO; gcs.max_ 
packet size - 64500; gcs.max throttle - 0.25; 

gcs.recv q hard limit - 9223372036854775807; gcs.recv q soft limit - 0.25; 
gcs.sync donor = NO; replicator.causal read timeout = PT30S; replicator.commit | 
order - 3 

Note] WSREP: Assign initial position for certification: -1, protocol version: 
=i 
Note] WSREP: Start replication 

Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:- 


Note] WSREP: protonet asio version 0 

Note] WSREP: backend: asio 

Note] WSREP: GMCast version 0 

Note] WSREP: (c3300902-b991-11e1-0800-040d7£7e9e14, 'tcp://0.0.0.0:4567') 

listening at tcp://0.0.0.0:4567 

Note] WSREP: (c3300902-b991-11e1-0800-040d7£7e9e14, 'tcp://0.0.0.0:4567') 

multicast: , ttl: 1 

Note] WSREP: EVS version 0 

Note] WSREP: PC version 0 

Note] WSREP: gcomm: connecting to group 'my wsrep cluster', peer ' 

Note] WSREP: view(view id (PRIM,c3300902-b991-11e1-0800-040d7f7e9e14,1) memb { 
c3300902-b991-11e1-0800-040d7f7e9e14, 

) joined { 

} left { 

} partitioned { 

p 


Note] WSREP: gcomm: connected 











Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636 
Note] WSREP: Shifting CLOSED -» OPEN (TO: 0) 

Note] WSREP: Opened channel 'my wsrep cluster' 

Note] /home/user/mysql/bin/mysqld: ready for connections. 

Version: '5.5.23' socket: '/tmp/mysql.sock' port: 3306 Source distribution, 
wsrep 23.6.r3755 

Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my idx = 0, 

memb num - 1 

Note] WSREP: Starting new group from scratch: 
C3315364-b991-11e1-0800-59a14f938404 

Note] WSREP: STATE EXCHANGE: sent state UUID: 
C3317547-b991-11e1-0800-4da9e025adff 

Note] WSREP: STATE EXCHANGE: sent state msg: 
C3317547-b991-11e1-0800-4da9e025adff 

Note] WSREP: STATE EXCHANGE: got state msg: 
c3317547-b991-11e1-0800-4da9e025adff from 0 (alpha) 

Note] WSREP: Quorum results: 











version = 2, 
PRIMARY, 
conf id = 0, 


component 


LU 
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members - 1/1 (joined/total), 
act id = 0; 

last appl. = -1, 

protocols = 0/4/2 (gcs/repl/appl), 


group UUID = c3315364-b991-11e1-0800-59a14f938404 
Note] WSREP: Flow-control interval: [8, 16] 
Note] WSREP: Restored state OPEN -> JOINED (0) 
Note] WSREP: Member 0 (alpha) synced with group. 
Note] WSREP: Shifting JOINED -> SYNCED (TO: 0) 
Note] WSREP: New cluster view: global state: 
€3315364-b991-11e1-0800-59a14£938404:0, 
view# 1: Primary, number of nodes: 1, my index: 0, protocol version 2 
Note] WSREP: wsrep notify cmd is not defined, skipping notification. 
Note] WSREP: Assign initial position for certification: 0, protocol version: 2 
Note] WSREP: Synchronized with group, ready for connections 








Note] WSREP: wsrep notify cmd is not defined, skipping notification. 


A further confirmation of the wsrep process in operation is the use of 
port 4567 on the server. In a production environment, if there are specific 
firewall rules for the MySQL port (e.g., 3306), then additional appropriate 
rules will be necessary for Galera Cluster communication. 


$ netstat -tulpn | grep -e 4567 -e 3306 
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 29329/mysqld 
tcp 0 0 0.0.0.0:4567 0.0.0.0:* LISTEN 29329/mysqld 


Incorrect MySQL Configuration Even with a defined wsrep cluster 
address, without the wsrep provider argument, MySOL will start 
without any issues; however, the expected features are not operational, and 
there is no indication of any problems. The MySQL error log will look like 
this: 

$ bin/mysqld safe --defaults-file-etc/my.cnf  --wsrep cluster address-gcomm:// & 


$ tail -20 data/^hostname' .err 


mysqld safe Starting mysqld daemon with databases from ... 


[Note] WSREP: Read nil XID from storage engines, skipping position init 
[Note] WSREP: wsrep load(): loading provider library 'none' 

[Note] /home/user/mysql/bin/mysqld: ready for connections. 

Version: '5.5.23' socket: '/tmp/mysql.sock' port: 3306 

Source distribution, wsrep 23.6.r3755 


State Snapshot Transfer (SST) 

While the initial configuration provides a minimum viable single MySOL 
instance running with Galera, this serves no real purpose and is not suffi- 
cient when adding Galera nodes to the cluster. The wsrep sst method 
variable is defined to ensure that when a new node is added (i.e., the 
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joiner), this can be initially synchronized with the known cluster (i.e., one 
node known as the donor). 

The diagram at http://www.codership.com/wiki/doku.php?id-state 
transfer protocol provides a great overview of the complexity of performing 
this initial snapshot and then synchronizing with the cluster. 

There are five possible values for the wsrep sst method configura- 
tion variable. These are rsync, rsync wan, xtrabackup, mysqldump, 
and skip.The current default is mysqldump; however, this may change in 
future releases. 

The mysqldump, rsync, and rsync_wan options are blocking opera- 
tions during the data acquisition stage on a joiner node and the donor. The 
rsync_wan option has further optimization for WAN transfers. The xtra- 
backup option provides the most flexibility; however, it requires addition- 
al software installation and configuration. 

Galera will, by default, use the IP address of etho for SST operations. In 
the example virtual environment being demonstrated, the correct IP is on 
eth1. The IP can be configured on all Galera nodes with the wsrep_sst_ 
receive address variable. For example: 

#SHOME/mysql/etc/my.cnf (on alpha) 


[mysqld] 
wsrep sst receive address-192.168.1.51 


rsync and rsync wan The rsync option is the easiest of the options 
available for SST. When set, and when applicable SSH keys are defined, a 
new joiner node can be easily synchronized: 
#SHOME/mysql/etc/my.cnf 


[mysqld] 
wsrep sst method-rsync 


mysqldump When using the mysqldump method (the current default), 
the additional wsrep_sst_auth option is required on both the joiner and 
donor nodes. This is a MySQL user account that must exist on all nodes. 
For example: 

#SHOME/mysql/etc/my.cnf 

[mysqld] 

wsrep sst method-mysqldump 

wsrep sst auth-galera:passwd 
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This introduces additional security issues. A new MySQL user is recom- 
mended because this requires network-level access (i.e., more than localhost) 
for the connection from the executed wsrep_sst_mysqldump command: 


mysql> GRANT ALL ON *.* TO galera@'192.168.1.%' IDENTIFIED BY PASSWORD 
'*59C70DA2F3E3A5BDFA46B68F5C8B8F25762BCCEFO0' ; 

This user requires the ability to remove and create the mysql schema 
for the joining node. For security purposes this account could be removed 
or disabled following use; however, SST may be required at any time if a 
new node is added or a level of corruption occurs and a resynchronization 
is necessary. This variable can technically be different between nodes, but 
would be an unnecessary complexity. 


NOTE While it would appear that rsync is a more practical method for SST, 
rsync works on a physical level while mysqldump works on a logical level. If 
the individual nodes have a different MySQL layout and options, for example, 
the datadiror innodb log file size, mysqldump is necessary. 


TIP Using identical MySQL configuration for all Galera Cluster nodes (aka the 
KISS principle) will remove unnecessary complexities, especially when trying 
to diagnose problems. 


Adding a Galera Node 

In order to add a second server to operate in a Galera Cluster, the installa- 
tion and configuration is identical. That is, install Galera, install MySOL 
with wsrep, install the MySQL starter database, and configure MySQL, 
including the definition of the applicable SST method. 

The only difference is the instantiation of the mysqld process, which 
requires a connection to one of the other known nodes in the cluster with 
the wsrep cluster address configuration option, in this example, the 
initial server, alpha: 
$ ./bin/mysqgld safe --defaults-file=etc/my.cnf \ 

--wsrep cluster address-gcomm://alpha & 

The initial handshake and SST transfer can be seen by viewing the re- 
spective error logs on both the joiner node (i.e., beta) and the selected 
donor node (i.e., alpha). 
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SST via rsync On the donor node (e.g., alpha), the following error log 
information confirms a successful SST request: 


Note] WSREP: New cluster view: global state: c3315364-b991-11e1-0800- 
59a14£938404:4, view# 100: Primary, number of nodes: 2, my index: 0, 
protocol version 2 

Note] WSREP: wsrep notify cmd is not defined, skipping notification. 
Note] WSREP: Assign initial position for certification: 4, protocol version: 2 
Note] WSREP: Node 1 (beta) requested state transfer from '*any*'. 
Selected O (alpha) (SYNCED) as donor. 

Note] WSREP: Shifting SYNCED -» DONOR/DESYNCED (TO: 4) 

Note] WSREP: wsrep notify cmd is not defined, skipping notification. 
Note] WSREP: Running: 'wsrep sst rsync 'donor' '192.168.1.52:4444/rsync sst' 
'galera:passwd' '/home/user/mysql/data/' '/home/user/mysqgl/etc/my.cnf' 
'c3315364-b991-11e1-0800-59a14f£938404' '4' 'Q'' 

Note] WSREP: sst donor thread signaled with 0 

Note] WSREP: Flushing tables for SST... 

Note] WSREP: Provider paused at c3315364-b991-11e1-0800-59a14f938404:4 
Note] WSREP: Tables flushed. 

Note] WSREP: Provider resumed. 

Note] WSREP: O (alpha): State transfer to 1 (beta) complete. 

Note] WSREP: Shifting DONOR/DESYNCED -» JOINED (TO: 4) 

Note] WSREP: Member 0 (alpha) synced with group. 

Note] WSREP: Shifting JOINED -» SYNCED (TO: 4) 

Note] WSREP: Synchronized with group, ready for connections 

Note] WSREP: wsrep notify cmd is not defined, skipping notification. 
Note] WSREP: 1 (beta): State transfer from 0 (alpha) complete. 

Note] WSREP: Member 1 (beta) synced with group. 








On the joiner node, you can see the following steps in the MySOL error 
log: 


€ A state difference is identified. 


e An Incremental State Transfer (IST) is first attempted but not 
successful due to a fundamental state difference. 


e A node is selected to operate in a donor role and a full SST is 
initiated on that node. 


e An InnoDB crash recovery is performed on retrieved state. 


e The node is recognized as in sync. 


[Note] WSREP: Shifting OPEN -» PRIMARY (TO: 4) 
[Note] WSREP: State transfer required: 

Group state: c3315364-b991-11e1-0800-59a14f938404:4 

Local state: 00000000-0000-0000-0000-000000000000:-1 
[Note] WSREP: New cluster view: global state: 
C3315364-b991-11e6e1-0800-59a14f938404:4, 
view# 100: Primary, number of nodes: 2, my index: 1, protocol version 2 
[Warning] WSREP: Gap in state sequence. Need state transfer. 
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[Note] WSREP: Running: 'wsrep sst rsync 'joiner' '192.168.1.52' '' 
'/home/user/mysql/data/' '/home/user/mysql/etc/my.cnf' '7285' 2>sst.err' 
[Note] WSREP: Prepared SST request: rsync|192.168.1.52:4444/rsync sst 
[Note] WSREP: wsrep notify cmd is not defined, skipping notification. 
[Note] WSREP: Assign initial position for certification: 4, protocol version: 2 
[Warning] WSREP: Failed to prepare for incremental state transfer: 

Local state UUID (00000000-0000-0000-0000-000000000000) 

does not match group state UUID (c3315364-b991-11e1-0800-59a14f938404): 1 
(Operation not permitted) 

at galera/src/replicator str.cpp:prepare for IST():439. IST will be unavailable. 
Note] WSREP: Node 1 (beta) requested state transfer from '*any*'. 
Selected O (alpha) (SYNCED) as donor. 

Note] WSREP: Shifting PRIMARY -» JOINER (TO: 4) 

Note] WSREP: Requesting state transfer: success, donor: 0 

Note] WSREP: O (alpha): State transfer to 1 (beta) complete. 

Note] WSREP: Member 0 (alpha) synced with group. 

Note] WSREP: SST complete, segno: 4 

nnoDB: The InnoDB memory heap is disabled 

nnoDB: Mutexes and rw locks use GCC atomic builtins 

nnoDB: Compressed tables use zlib 1.2.3 

nnoDB: Using Linux native AIO 

nnoDB: Initializing buffer pool, size - 128.0M 

nnoDB: Completed initialization of buffer pool 

nnoDB: highest supported file format is Barracuda. 

nnoDB: The log sequence number in ibdata files does not match 

nnoDB: the log sequence number in the ib logfiles! 

InnoDB: Database was not shut down normally! 

nnoDB: Starting crash recovery. 

nnoDB: Reading tablespace information from the .ibd files... 

nnoDB: Restoring possible half-written data pages from the doublewrite 
nnoDB: buffer... 

nnoDB: Waiting for the background threads to start 

nnoDB: 1.1.8 started; log sequence number 1596023 

Note] Event Scheduler: Loaded 0 events 

Note] WSREP: Signalling provider to continue. 

Note] WSREP: Received SST: c3315364-Db991-11e1-0800-59a14f938404:4 

Note] WSREP: SST received: c3315364-Db991-11e1-0800-59a14f938404:4 

Note] /home/user/mysql/bin/mysqld: ready for connections. 

Version: '5.5.23' socket: '/tmp/mysql.sock' port: 3306 Source distribution, 
wsrep 23.6.r3755 

Note] WSREP: 1 (beta): State transfer from 0 (alpha) complete. 

Note] WSREP: Shifting JOINER -» JOINED (TO: 4) 

Note] WSREP: Member 1 (beta) synced with group. 

Note] WSREP: Shifting JOINED -> SYNCED (TO: 4) 

Note] WSREP: Synchronized with group, ready for connections 











Operation Confirmation 

Galera provides a number of MySQL status variables to view the running op- 
eration of the cluster. Following this initial cluster creation and adding a sec- 
ond node, the values in this example are as shown in the following sections. 
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alpha Status 
alpha» SHOW GLOBAL STATUS LIKE 'wsrep$'; 
hoe ee cese I UE hae Se Ss E LL E LE E EE * 
Variable name Value 
ao eS eee ee E ELE JEDE * 
wsrep local state uuid 88bdf845-b0a4-11e1-0800-4dbl1faf4581b 
wsrep protocol version 4 
wsrep last committed 0 
wsrep replicated 0 
wsrep replicated bytes 0 
wsrep received 6 
wsrep received bytes 376 
wsrep local commits 0 
wsrep local cert failures 0 
wsrep local bf aborts 0 
wsrep local replays 0 
wsrep local send queue 0 
wsrep local send queue avg 0.500000 
wsrep local recv queue 0 
wsrep local recv queue avg 0.000000 
wsrep flow control paused 0.000000 
wsrep flow control sent 0 
wsrep flow control recv 0 
wsrep cert deps distance 0.000000 
wsrep apply oooe 0.000000 
wsrep apply oool 0.000000 
wsrep apply window 0.000000 
wsrep commit oooe 0.000000 
wsrep commit oool 0.000000 
wsrep commit window 0.000000 
wsrep local state 4 
wsrep local state comment Synced (6) 
wsrep cert index size 0 
wsrep causal reads 0 
wsrep cluster conf id 2 
wsrep cluster size 2 
wsrep cluster state uuid 88bdf£845-b0a4-11e1-0800-4db1faf4581b 
wsrep cluster status Primary 
wsrep connected ON 
wsrep local index 0 
wsrep provider name Galera 
wsrep provider vendor Codership Oy <info@codership.com> 
wsrep provider version 23.2.1(r129) 
wsrep ready ON 
RENE oH SSS Se SSRs Ree eee SS PESAS eo eee ae eS ee Re ea + 
39 rows in set (0.00 sec) 
beta Status 
beta» SHOW GLOBAL STATUS LIKE 'wsrep$'; 
Pease SSSR Sa MURUS EU CMS FASAS REHE REAA RAEE ALMUS BS IR SINUS RIS * 


| Variable name | Value 
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wsrep local state uuid 
wsrep protocol version 
wsrep last committed 
wsrep replicated 

wsrep replicated bytes 
wsrep received 

wsrep received bytes 
wsrep local commits 

wsrep local cert failures 
wsrep local bf aborts 
wsrep local replays 
wsrep local send queue 
wsrep local send queue 
wsrep local recv queue 
wsrep local recv queue avg 
wsrep flow control paused 
wsrep flow control sent 
wsrep flow control recv 
wsrep cert deps distance 
wsrep apply oooe 

wsrep apply oool 

wsrep apply window 

wsrep commit oooe 

wsrep commit oool 

wsrep commit window 

wsrep local state 

wsrep local state comment 
wsrep cert index size 
wsrep causal reads 

wsrep cluster conf id 
wsrep cluster size 

wsrep cluster state uuid 
wsrep cluster status 

wsrep connected 

wsrep local index 

wsrep provider name 

wsrep provider vendor 
wsrep provider version 
wsrep ready 


avg 

















88bdf845-b0a4-11e1-0800-4db1faf4581b 


Mo 
œ 


23333333 


.000000 
.000000 


.000000 
.000000 
.000000 
.000000 
.000000 
.000000 
.000000 


HN. OO .O O0 O0 OO O O O O O OO O OO O H uw oO Oo Oo E 


Synced (6) 

0 

0 

2 

2 
88bdf845-b0a4-11e1-0800-4db1faf4581b 
Primary 

ON 

1 

Galera 

Codership Oy <info@codership.com> 
23.2.1(r129) 

ON 


The Galera Wiki at http://www.codership.com/wiki/doku.php?id- 


monitoring provides more information on important status variables and 


their respective values for checking the cluster integrity, node status, repli- 


cation health, and slow performance bottlenecks. 


We can now run a simple test, using the same test steps as described in 


the appendix, for testing replication. This will create a table and use a sim- 


ple stored procedure to simulate some load. We can confirm the data is 


consistent in the table with the following simple test: 
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alpha» PAGER md5sum 

alpha» SELECT * FROM numbers ORDER BY id; 
bb39b97bfe7e28b8abdbf9b943b0cd3a - 
1048576 rows in set (0.31 sec) 

alpha» NOPAGER 

beta» PAGER md5sum 

beta» SELECT * FROM numbers ORDER BY id; 
bb39b97bfe7e28b8abdbf9b943b0cd3a - 
1048576 rows in set (0.40 sec) 

beta» NOPAGER 


In addition we can now see changes in the MySQL Status variables: 


beta» SHOW GLOBAL STATUS LIKE 'wsrep$'; 


PS Se see SS r foe SSeS Se ee + 
Variable name Value 

o-oo SSS sae See nnna ea Hoes See r a eee MEUS + 
wsrep local state uuid 88bdf845-b0a4-11e1-0800-4dbl1faf4581b 
wsrep protocol version 4 
wsrep last committed 47 
wsrep replicated 0 
wsrep replicated bytes 0 
wsrep received 69 
wsrep received bytes 42700075 











Under these normal testing procedures, no addition wsrep information 
is written to the MySQL error log. In general, after the initial creation and 
SST-related log messaging in the error log, any later wsrep messages 
should be considered an error to investigate. 


Multi-Master Replication 

Galera provides true active multi-master replication by default. There is no 
additional configuration necessary. We can repeat the same test in a differ- 
ent schema for verification: 


beta» CREATE SCHEMA IF NOT EXISTS test2; 
beta» USE test2; 
beta» SOURCE fill numbers.sqgl 
We can also verify several different status variable changes to indicate 


the local operations. For example: 


beta» SHOW GLOBAL STATUS LIKE 'wsrep%'; 


wsrep replicated 27 
p rep 
| wsrep replicated bytes | 29514748 
wsrep received 81 
p. 
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| wsrep received bytes | 56931329 
| wsrep local commits | 21 


In comparison, the before values were: 


| wsrep replicated | o 
| wsrep replicated bytes | 0 
| wsrep received | 69 

| wsrep received bytes | 42700075 
| lo 


wsrep local commits 


Optimal MySOL Configuration 
There are several recommended configuration settings in addition to those 
that have already been discussed. The full list of recommendations includes 


e wsrep provider This is a path and filename of the Galera library. 
This is a required parameter. 


e wsrep cluster address Thisis a gcomm:// specific address to 
one node of the cluster. This is a required parameter. 


e wsrep cluster name This is a name given to the cluster in use. 


e wsrep sst method This parameter defines the method to 
perform an initial state snapshot transfer (SST). The current default 
value when not specified is mysqldump. The recommended setting is 
rsync or xtrabackup. 


e wsrep node address This is the IP address of the node. When 
not specified, this will default to the first retrievable IP address. This 
may not be ideal for various network configurations that have 
multiple network connections. 


e wsrep node name This is the human readable name of the node. 
This defaults to the hostname if not specified. 


e wsrep slave threads This specifies the number of parallel slave 
threads. The recommended value is 4 * cores. 


e wsrep provider options This option specifies various other 
options. It is recommended that the gcache size is set 
appropriately to maximize throughput. 


A. default installation shows a large number of possible values for 
wsrep provider options.These currently include 
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mysql» SHOW GLOBAL VARIABLES LIKE 'wsrep provider options' VG 
base host - 192.168.1.51; 

base port - 4567; 

evs.debug log mask = 0x1; 

evs.inactive check period = PT0O.5S; 


evs.inactive timeout = PT15S; 
evs.info log mask = 0; 
evs.install timeout = PT15S; 


evs.join retrans period = PT0.3S; 
evs.keepalive period = PT1S; 
evs.max install timeouts = 1; 
evs.send window - 4; 
evs.stats report period = PT1M; 

evs.suspect timeout - PT5S; 

evs.use aggregate - true; 

evs.user send window - 2; 

evs.version - 0; 

evs.view forget timeout - PT5M; 

gcache.dir = /home/user/mysql/data/; 
gcache.keep pages size = 0; 

gcache.mem size - 0; 

gcache.name = /home/user/mysql/data//galera.cache; 
gcache.page size = 128M; 

gcache.size = 128M; 

gcs.fc debug = 0; 

gcs.fc factor = 0.5; 

gcs.fc limit - 16; 

gcs.fc master slave - NO; 

gcs.max packet size - 64500; 

gcs.max throttle - 0.25; 

gcs.recv q hard limit - 9223372036854775807; 
gcs.recv q soft limit - 0.25; 

gcs.sync donor - NO; 

gmcast.listen addr = tcp://0.0.0.0:4567; 
gmcast.mcast addr - ; 

gmcast.mcast ttl - 1; 

gmcast.peer timeout = PT3S; 

gmcast.time wait = PT5S; 

gmcast.version = 0; 

ist.recv_addr = 192.168.1.51; 

pc.checksum = true; 

pc.ignore quorum - false; 

pc.ignore sb - false; 

pc.linger - PT2S; 

pc.npvo - false; 

pc.version - 0; 

protonet.backend - asio; 

protonet.version - 0; 
replicator.causal read timeout = PT30S; 
replicator.commit order - 3 








Adding Nodes 
To show the true strengths of this technology, we can add a third node to 
the cluster following the same steps when adding a second node (e.g. 
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gamma). After confirmation in the error log that the node is correctly syn- 
chronized, schema and data are confirmed: 


$ cd mysql 

$ tail data/^hostname'.err 

120607 16:19:02 [Note] WSREP: 2 (gamma): State transfer from 0 (alpha) complete. 
120607 16:19:02 [Note] WSREP: Shifting JOINER -» JOINED (TO: 75) 

120607 16:19:02 [Note] WSREP: Member 2 (gamma) synced with group. 

120607 16:19:02 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 75) 

120607 16:19:02 [Note] WSREP: Synchronized with group, ready for connections 

$ bin/mysql -uroot -e "SHOW SCHEMAS" 


information schema 
mysql 
performance_schema 
test 








5 rows in set (0.00 sec) 
$ bin/mysql -uroot 
gamma» PAGER md5sum 
gamma» SELECT * FROM book3.numbers ORDER BY id; 
bb39b97bfe7e28b8abdbf9b943b0cd3a - 
1048576 rows in set (0.34 sec) 
gamma» NOPAGER 
We can complete the test by repeating our example procedure and data 
in a new test3 schema and confirming that the first two nodes receive all 


data. 


Additional Features 

In addition to the already discussed features, several other features are 
important for production use and implementation. These include the 
following: 


e Galera provides an independent arbitrator (garbd daemon) that 
operates as a dataless Galera node. The garbd node can help in 
detecting and dealing with network splits in an optimal way. A 
well-positioned garbd node can prevent split-brain situations. 


e Galera supports SSL communications, which is important for added 
security, especially in cloud deployments. This is defined with 
the options socket.ssl_cert and socket.ssl_ key. For the 
initial SST, additional precautions are necessary depending on the 
applicable method used. 
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e SST enables customizable methods for obtaining the initial dataset 
for a new node. 


Online rolling schema upgrades are possible by altering execution 
method from Total Order Isolation (TOI) to Rolling Schema Upgrade 
(RSU). While the default TOI method is predictable and guarantees 
data consistency, it does limit the cluster's high availability, e.g., during 
long running ALTER statements. There are several caveats for effective 
operation with this feature as the result of a blocking ALTER statement 
under normal MySQL operations. 


Galera Cluster Implementation Recommendations 

It is highly recommended that you run more than two Galera nodes in a 
production environment, or that you ensure you use a separate Galera 
Arbitrator. The garbd daemon is designed to avoid a split-brain situation 
with a minimum of two servers. More information can be found at http:// 
www.codership.com/wiki/doku.php?id-galera arbitrator. 

The mysqld safe wrapper daemon is not the ideal manager for handing 
an automatic MySQL restart in a cluster situation. Future work is necessary 
here with your production environment to support various disaster situations. 

Installation of MySQL and Galera should not be in a /home/user account 
ona production system as demonstrated in this working example. Applicable 
software release procedures and appropriate service startup and shutdown 
steps are also needed. 

The FromDual Performance Monitor for MySOL (MPM) written by Oli 
Sennhauser provides monitoring and graphing of the important triggers of 
Galera Cluster for MySQL. These monitors are available with Zabbix, an 
open source monitoring tool. More information about MPM can be found 
at http://fromdual.com/mpm-0-9-is-out. 

There is no information on when MySOL monitoring plugins for other 
available common monitoring will become available. Every production 
system should include adequate monitoring and alerting. 


Percona XtraDB Cluster 


Percona offers a MySQL server deployment with Galera automatically in- 
cluded. This provides the Percona Server software that includes additional 
instrumentation and performance patches and the XtraDB storage engine, 
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which is a Percona variant of InnoDB. More details can be found at http:// 
www.percona.com/software/percona-xtradb-cluster. 


MariaDB Galera Cluster 


Work is occurring to have support for Galera in MariaDB. More informa- 
tion can be found at the Monty Program Knowledge Base found at http:// 
kb.askmonty.org/en/what-is-mariadb-galera-cluster/. 


Galera Wrap-Up 


The information in this chapter is only an introduction to Galera Cluster by 
the team at Codership. Details of server monitoring, performance analysis 
and tuning, arbitration management and handling node failures, recovery, 
and provisioning are important areas that require far greater explanation 
to appreciate the available features of this product. Additional information, 
including benchmarks, new features, presentations, and blogs, can be 
found at http://www.codership.com/. 


Getting More Help 
Galera provides a discussion mailing list for any questions or issues. 
Details can be found at http://codership.com/info/mailing-list. 


Tungsten Replicator 


Tungsten Replicator provides an extensive list of additional replication fea- 
tures to MySQL, including supporting seamless failover of MySQL serv- 
ers, flexible filtering of operations, multi-master and multi-master to slave 
support (i.e., fan in), parallel replication, and much more. 

In addition to MySQL replication, Tungsten Replicator can provide het- 
erogeneous data management with other RDBMS and noSQL products, 
including data synchronization between MySQL and Oracle and vice versa. 

Tungsten Replicator is available under the open source GNU GPL v2 
license. Continuent, the creators of Tungsten Replicator, also provide com- 
mercial support with an enterprise offering. More information can be found 
at http://www.continuent.com/solutions/overview. 
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Features 


There is a long list of possible features that can be discussed. In this sec- 
tion, which is an introduction to Tungsten Replicator, the following will be 
discussed and demonstrated: 


e Master failover via slave promotion 

e Bidirectional replication (e.g., active/active multi-master) 

e Parallelization and replication prefetching for improved performance 
e Complex topologies 


With traditional MySQL replication, the version of a MySQL slave 
should be the same or greater than that of the master. With Tungsten Rep- 
licator, this limitation does not exist. It is possible to replicate from MySQL 
5.5 to MySQL 5.0, for example. There are limitations to any features that are 
in the newer version of MySQL; however, Tungsten can handle the error 
condition gracefully and not interrupt replication from continuing. 


NOTE Tungsten Replicator has one significant benefit over other products. 
There is no change to the standard MySQL installation necessary. There are no 
custom or patched MySQL binaries to install or maintain. Tungsten Replicator 
works with your existing MySQL environment and can replicate between 
MySQL 4.1, 5.0, 5.1, 5.5, and 5.6 as well as different flavors, including 
MySQL Community, MySQL Enterprise, MariaDB, and Percona Server. 


References 


The current version of Tungsten Replicator available at the time of this 
publication is 2.0.5.You can find additional information, including the most 
current version, at the following sites: 


e Product Page http://www.continuent.com/solutions/tungsten- 
replicator 


e Downloads http://tungsten-replicator.org 


e Documentation http://code.google.com/p/tungsten-replicator/w 


"Tungsten supports replicating to MySQL 4.1, but not from it. 
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Prerequisites 


Tungsten Replicator has a number of operating system, network, software, 
and MySQL prerequisites. These must be installed on all servers that will 
be used in the Tungsten cluster. The following steps are applicable for an 
Ubuntu/Debian operating system. Similar statements are available for Red 
Hat/CentOS/Oracle Linux distributions. 

$ sudo apt-get install -y ruby 

$ which ruby 

$ ruby --version 

$ which rsync 

$ echo "p 'hello'" | ruby -ropenssl 

$ sudo apt-get install openjdk-6-jre 

$ java -version 

$ uname -n 

$ hostname --ip-address 


CAUTION An important prerequisite is that all hosts resolve to a correct 
IP address and not a 127. X.X.X loopback address. The hostname --ip- 
address command can be used to confirm this. 


If Tungsten is used for backup and restore management, sudo access 
without a password is also necessary. 

More information on these prerequisites can be found at https://docs 
.continuent.com/wiki/display/TEDOC/System+Requirements. 


Installation with Tungsten Sandbox 


The easiest way to demonstrate the features of Tungsten Replicator is in a 
MySQL Sandbox environment. Refer to the MySQL Sandbox installation 
instructions in the appendix to configure the necessary Sandbox software 
first. The following steps will then install Tungsten Replicator and Tungsten 
Sandbox. 


TIP Tungsten Replicator is a complex product with several administration tools. 
The Tungsten Sandbox provides an easy way to evaluate and review these tools 
and the respective output. You can find a good cheat sheet for understanding 
the tungsten operations in comparison to MySQL replication. Details can be 
found at http://code.google.com/p/tungsten-replicator/wiki/Cheat_Sheet. 
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$ cd SHOME/sandboxes 

4 Get Current version from http://bit.ly/tr20 builds 

$ curl --silent -o tungsten-replicator.tar.gz https://s3.amazonaws. 
com/files.continuent.com/builds/nightly/ 
tungsten-2.0-snapshots/tungsten-replicator-2.0.6-683.tar.gz 

$ tar xvfz tungsten-replicator.tar.gz 

rm -f tungsten-replicator.tar.gz 

mv tungsten-replicator-* tungsten-replicator 

cd tungsten-replicator 

curl --silent -o tungsten-sandbox 
http://tungsten-toolbox.googlecode.com/files/tungsten-sandbox-2.0.11 
chmod «x tungsten-sandbox 

./tungsten-sandbox --help 

mkdir SHOME/tsb2 


$ 
$ 
$ 
$ 


V Ur Ur 


Finally, you can create a Tungsten cluster in a Tungsten Sandbox envi- 
ronment with a single command. The following command will create a 
master-master cluster: 


$ cd S$HOME/sandboxes/tungsten-replicator 

$ mkdir -p $HOME/tsandboxes/master-master 

$ ./tungsten-sandbox --topology bi-dir -1 12300 -r 10300 

-t SHOME/tsandboxes/master-master -m 5.5.24 -p 7300 -d tsb-mm 
The options specified relate to: 


e --topology The type of Tungsten topology. This includes bi-dir, 
direct, star, all-masters, fan-in, and master-slave. 


e -1 TheTHL service port number. 
e -r The RMI service port number. 


e -t The full path to the Tungsten Replicator administration 
directory. 


e -m Refers to the MySQL version to be used. 

e -p Refers to the base port number for the MySQL nodes. 

e -d The relative path to the MySQL Sandbox installed MySQL 
topology. 


TIP Ifyou add the - -verbose option for Tungsten Sandbox, you will get more 
detailed information on the commands used to install the cluster. 


You can get a full list of Tungsten Sandbox options with the help option: 


$ ./tungsten-sandbox -h 
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There are two installed components of the new Tungsten Sandbox. The 
first are MySQL nodes, which are installed in SHOME/sandboxes/tsb-mm 
directory and operate like a regular sandbox environment having applica- 
ble tools to start, stop, and access MySOL. The second component is the 
Tungsten Replicator configuration, which is installed in $SHOME/tsandbox- 
es/master-master asspecified by -t. 


TIP Ifyou receive an error when creating a Tungsten Sandbox, you may have 
created a port conflict with another MySQL Sandbox or Tungsten Sandbox 
installation. This can be easily confirmed by looking for any existing MySQL 
and Java processes that are running with $ ps -ef | grep -e mysql 


-e java. 


Reviewing a Tungsten Replicator Environment 

The following administration steps will give you an overview of some of 
the Tungsten Replicator functionality that can be used to review and ad- 
minister a Tungsten cluster. 


Overall Status of Cluster Services 


$ cd SHOME/tsandboxes/master-master 
./replicator all status 
#1 
Tungsten Replicator Service is running (PID:31805). 
#2 
Tungsten Replicator Service is running (PID:32717). 
The available options for replicator all (a wrapper to the repli- 
cator command) are start, stop, restart, condrestart, status, 
dump, and console. 


Important File Locations You can find the configuration files for each 
Tungsten node in the respective db1 and db2 sub-directories. For example, 
with db1: 
$ cd SHOME/tsandboxes/master-master 
$ cat db1/configs/tungsten.cfg 

The tungsten.cfg file contains a JavaScript Object Notation (JSON) 
representation of the topology installed. This is also used by the installer 
tools to perform updates to the cluster configuration. 
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Additional configuration files can be found in db1/releases/tung- 
sten-replicator/tungsten-replicator/conf.The important files are 


e wrapper.conf This is the configuration for the Java instance and 
is used to assign resources or enable debugging. 


e static-SERVICE-NAME.properties This contains information 
for each service that is configured. There are currently over 200 
settings. 


Tungsten Sandbox provides a shortcut to examine the node configura- 
tion files: 
$ cd SHOME/tsandboxes/master-master 
$ ./db1/show_conf 

Tungsten Replication supports three types of logs. These are the Master 
Relay Logs (MRL), the Transaction History Logs (THL), and the Service 
Logs. The Master Relay Logs are used when the master is extracting events 
from the MySQL server. The Transaction History Logs are used to store 
events after they are extracted. Finally the service logs are the replicator 
operating logs. 

A very important feature of the THL is that Tungsten adds a global trans- 
action ID when it extracts data from binary logs. This is a fundamental differ- 
ence from native replication, as it allows seamless failover without manual 
inspection of the binary logs. 

The service logs are: 


$ cat dbl/releases/tungsten-replicator/tungsten-replicator/log/trepsvc.log 
$ cat dbl/releases/tungsten-replicator/tungsten-replicator/log/user.log 

When using Tungsten Sandbox there is also a shortcut to viewing the 
logs easily: 
$ cd SHOME/tsandboxes/master-master 
$ ./dbl/show log 

It is important to understand and monitor all of these logs (MRL, THL, 
and Service) as they are one cause of additional diskspace usage. For ex- 
ample, to simulate an expire logs days-4 you would set replica- 
tor.store.thl.log file retention-4d in the configuration file. 
More information on these logs and how to manage and configure usage 
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can be found at http://code.google.com/p/tungsten-replicator/wiki/ 
TRCAdministration#Managing_replicator_log space. 


Testing the Tungsten Sandbox Generally, you would first use the 
various administration tools to check and verify the operation of the cluster 
as later documented. Running the following included test script helps 
provide the following command examples with meaningful results to 
display: 

$ cd SHOME/tsandboxes/master-master 


./test topology 
# Testing topology bi-dir with 2 nodes. 





# Master nodes: [1 2] - Slave nodes: [1 2] 
# node 1 

ok - Tables from all masters 
ok - Views from all masters 
ok - Records from master #1 
ok - Records from master #2 
ok - Node #1-alpha online 

ok - Node #1-bravo online 

# node 2 

ok - Tables from all masters 
ok - Views from all masters 
ok - Records from master #1 
ok - Records from master #2 
ok - Node #2-alpha online 

ok - Node #2-bravo online 

d 2 


Details of Individual Node Servers You can view more information 
about the node servers in the given Tungsten cluster by looking at the 


services: 

$ ./trepctl all services (also executed with ./services all) 
#1 

Processing services command... 

NAME VALUE 


appliedLastSeqno: 8 





appliedLatency : 0.303 
role : master 
serviceName : alpha 
serviceType : local 
started : true 
state : ONLINE 





NAME VALUE 
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appliedLastSeqno: 8 





appliedLatency 1.211 
role Slave 
serviceName bravo 
serviceType remote 
started true 
state : ONLINE 
Finished services command... 
#2 


The Tungsten Toolbox (http://code.google.com/p/tungsten-toolbox/) 
also provides the following additional tools to help interpret the Tungsten 
status output: 


e simple services This filters the output of trepct1 services to 
produce a compact summary. 


e trepctl-progress Shows how much work a replicator has done 
and how much it needs to do to process its THL files. 


To delve into more specifics for a given Tungsten node you can execute 
the following trepct1 on an individual given node: 


$ dbl/trepctl -service alpha status 
Processing status command... 





NAME VALUE 
appliedLastEventId mysql-bin.000002:0000000000001551;0 
appliedLastSeqno : 8 

appliedLatency 0.303 

clusterName default 

currentEventId mysql-bin.000002:0000000000001551 
currentTimeMillis 1340401306138 
dataServerHost 127.0 Oil 

extensions : 

latestEpochNumber «0 

masterConnectUri thl://:/ 
masterListenUri thl://127.0.0.1:12300/ 
maximumStoredSeqNo 8 

minimumStoredSeqNo 0 

offlineRequests NONE 

pendingError NONE 

pendingErrorCode NONE 
pendingErrorEventId NONE 

pendingErrorSeqno oad 
pendingExceptionMessage: NONE 
resourcePrecedence 99 

rmiPort 10300 

role master 


seqnoType 


java.lang.Long 
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serviceName alpha 
serviceType local 
simpleServiceName alpha 
siteName default 
sourceId 127.0.0.1 
state ONLINE 
timeInStateSeconds 2035.975 
uptimeSeconds 2040.584 


Finished status command... 


Other syntax that should be reviewed includes 


$ dbl/trepctl 
$ dbl/trepctl 
$ dbl/trepctl 


-service alpha status -name tasks 
-service alpha status -name shards 
-service alpha status -name stores 


The -name variants are best used to provide information for parallel 
replication or to provide information for huge row-based replication (RBR) 
transactions by providing details of the chunks processed. The output is 
not shown here due to space limitations. To see the full options of the 


Tungsten Replicator Control Utility, you can view the command help with: 
$ dbl/trepctl help 


More information on the Tungsten Replicator Control Utility can be 
found at https://docs.continuent.com/wiki/display/ TEDOC/The- Tungsten 
+Replicator+Control+Utility+ %28trepctl%29. 


Transaction History Logs (THL) The Transaction History Logs (THL) 
hold all the data that is taken from the MySQL master binary logs and used 
by all Tungsten Replicators. The thl command can provide information 
about these logs, including: 


db1/thl -service alpha info 
NFO thl.log.DiskLog Using directory '/home/user/tsandboxes/master- 


master/dbl/tlogs/alpha/' for replicator logs 


NFO thl.log.DiskLog Checksums enabled for log records: true 
NFO thl.log.DiskLog Using read-only log connection 
NFO thl.log.DiskLog Loaded event serializer class: 


com.continuent.tungsten.replicator.thl.serializer.ProtobufSerializer 
NFO thl. 
/home/user/tsandboxes/master-master/db1/tlogs/alpha 

NFO thl.log.LogIndex Constructed index; total log files added=1 
- main NFO thl.log.DiskLog Validating last log file: 
/home/user/tsandboxes/master-master/db1/tlogs/alpha/thl.data.0000000001 


og.LogIndex Building file index on log directory: 


- main 











- main NFO thl.log.DiskLog Setting up log flush policy: 
fsyncIntervalMillis-0 fsyncOnFlush-false 
- main NFO thl.log.DiskLog Idle log connection timeout: 28800000ms 
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[- main] INFO thl.log.DiskLog Log preparation is complete 
min seq# = 0 
max seq# = 8 
events = 8 
The following option will list the details within the given log: 
$ db1/thl -service alpha list -seqno 5 
SEQH = 5 / FRAGH = 0 (last frag) 
- TIME = 2012-06-22 17:10:17.0 
- EPOCH# = 0 
- EVENTID = mysql-bin.000002:0000000000001003;0 
- SOURCEID = 127.0.0.1 
- METADATA = [mysql server id-101;unsafe for block commit; 
service-alpha;shard-test] 
- TYPE - com.continuent.tungsten.replicator.event.ReplDBMSEvent 
- OPTIONS = [##charset = UTF-8, autocommit = 1, sql auto is null = 0, 
foreign key checks = 1, unique checks = 1, sql mode = '', character set client = 
33, collation connection - 33, collation server - 8] 
- SCHEMA - 
- SQL(0) = DROP TABLE IF EXISTS TUNGSTEN INFO.alpha, ^test^.^v1^ /* 


$ dbl/thl -service alpha list -seqno 6 

SEQ# = 6 / FRAG# = 0 (last frag) 

- TIME = 2012-06-22 17:10:17.0 

- EPOCH# = 0 

- EVENTID = mysql-bin.000002:0000000000001137;0 

- SOURCEID = 127.0.0.1 

- METADATA = [mysql server id-101;unsafe for block commit; 
service-alpha;shard-test] 

- TYPE - com.continuent.tungsten.replicator.event.ReplDBMSEvent 

- OPTIONS = [##charset = UTF-8, autocommit = 1, sql auto is null = 0, 


foreign key checks = 1, unique checks = 1, sql mode = '', 
character set client - 33, collation connection - 33, collation server 
- SCHEMA - 

- SQL(0) = create table test.t1(i int not null primary key, c char(20)) 
engine- innodb /* | SERVICE = [alpha] */ 


The commands that can be performed with th1 include list, index, 
purge, and info. More information about the Transaction History Log 
Utility can be found at https://docs.continuent.com/wiki/display/TEDOC/ 


The+Transaction+History+Log+Utility+ %28thl%29. 


Tungsten Sandbox Cleanup The Tungsten Sandbox includes a number 


of working components. You can easily clean up all installed software with 


a single command. For example: 


$ cd SHOME/tsandboxes/master-master 
$ ./erase tsandbox 
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Other Documentation Additional documentation can be found at http:// 
code.google.com/p/tungsten-toolbox/wiki/TungstenSandbox. Detailed docu- 
mentation for explaining how to install many different topologies of Tungsten 
Replicator can be found at http://code.google.com/p/tungsten-replicator/ 
wiki/TungstenReplicatorCookbook. Details of administrative functions can 
be found at http://code.google.com/p/tungsten-replicator/wiki/TRCAdmin 
istration#Administration. 


Other Tungsten Sandbox Examples 
The following command will create a star Tungsten cluster: 


$ mkdir -p SHOME/tsandboxes/star 

$ cd SHOME/sandboxes/tungsten-replicator 

$ ./tungsten-sandbox --topology star -1 12400 -r 10400 -n 5 --hub 3 
-t SHOME/tsandboxes/star -m 5.5.24 -p 7400 -d tsb-star --verbose 


The following command will create a fan-in cluster: 


$ mkdir -p $HOME/tsandboxes/fan-in 
$ cd S$HOME/sandboxes/tungsten-replicator 
$ ./tungsten-sandbox --topology fan-in -1 12500 -r 10500 -n 3 --fan-in 
3 -t SHOME/tsandboxes/fan-in -m 5.5.24 -p 7500 -d tsb-fi --verbose 
Be sure to repeat the Tungsten Replicator administration commands 


shown here to observe the difference in the various configurations. 


Manual Tungsten Installation 


The following steps will install a Tungsten Replicator configuration in the 
test virtual environment that is defined in the appendix. For the following 
examples the alpha, beta, and gamma servers will be used. 


MySQL Setup 
MySQL must first be installed and operating. Refer to the appendix for the 
basic installation of MySOL on the given server. 


MySQL Configuration 

Tungsten Replicator requires and recommends the following additional 
MySQL configuration settings: 

# SHOME/mysql/etc/my.cnf 


[mysqld] 
server-id=51 
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log-bin=mysql-bin 
default-storage-engine=InnoDB 

# Recommended 

innodb flush log at trx commit-2 
max allowed packet-48M 





NOTE Tungsten relies on the MySQL binary log for replication. You should use 
the appropriate practices that best suit your business continuity needs and 
hardware capabilities, including configuring sync binlogand innodb 
flush log at trx commit appropriately. 





The MySQL instance can now be restarted: 


cd SHOME/mysql 

./bin/mysqladmin shutdown 

./bin/mysqgld safe --defaults-file=etc/my.cnf & 
tail -f data/^hostname' .err 


Ur Ur Ur Ur 


These steps should be completed on all servers that you wish to include 
in the Tungsten cluster. In addition, a MySOL user with root-level privi- 
leges is necessary. This should be appropriately secured to minimize unau- 
thorized access. There are no naming requirements on the username, i.e., 
tungsten is used here only for reference. 


mysql> GRANT ALL ON *.* TO tungsten9'192.168.1.$' IDENTIFIED BY 'continuent' 
-» WITH GRANT OPTION; 


Failure to do so will cause errors in installation, including: 


ERROR »» alpha »» Unable to connect to the MySQL server using 
root@alpha:3306 (WITH PASSWORD) 


or 


ERROR »» alpha »» The database user is missing some privileges or 
the grant option. 
Run 'mysql -u -p -h -e "GRANT ALL ON *.* to tungsten@alpha WITH GRANT OPTION"' 


Tungsten Replicator Installation 

The following steps will download Tungsten Replicator. Refer to the down- 
load link for the most current version available. You can also download 
regular daily builds from http://bit.ly/tr20 builds. 

$ cd /tmp 


$ wget http://tungsten-replicator.googlecode.com/files/ 
tungsten-replicator-2.0.5.tar.gz 
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tar xvfz tungsten-replicator*.tar.gz 

rm -f tungsten-replicator*.tar.gz 

mv tungsten-replicator-* tungsten-replicator 
cd tungsten-replicator 


Ur Ur Ur Ur 


Tungsten Replicator Master/Slave Setup 

With the installation and operation of MySOL on the necessary servers, 
and the installation of Tungsten Replicator on the server, the following sin- 
gle command will configure the specified MySOL topology: 


$ TUNGSTEN HOME-$HOME/tungsten 

$ MASTER-alpha 

$ SLAVEl-beta 

$ SLAVE2-gamma 

$ ./tools/tungsten-installer \ 
--master-slave --master-host-$MASTER \ 
--datasource-user-tungsten --datasource-password-continuent N 
--datasource-log-directory-$HOME/mysql/data \ 
--datasource-mysql-conf-$HOME/mysql/etc/my.cnf \ 
--service-name=effectivemysql \ 
--home-directory-$TUNGSTEN HOME \ 
--cluster-hosts=$MASTER, $SLAVE1,$SLAVE2 \ 
--Start-and-report 


This takes a few moments to do all the necessary checking and verification: 


INFO >> alpha >> Getting services list 
INFO >> alpha >> ...... 

Processing services command... 

NAME VALUE 


appliedLastSeqno: 0 


appliedLatency : 0.832 

role : master 
serviceName : effectivemysql 
serviceType : local 

started : true 

state : ONLINE 


Finished services command... 

INFO >> beta >> Getting services list 
INFO >> beta >> 

Processing services command... 

NAME VALUE 
appliedLastSeqno: 0 

appliedLatency : 9.995 

role : Slave 
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serviceName : effectivemysql 
serviceType : local 

started : true 

state : ONLINE 


Finished services command... 

INFO >> gamma >> Getting services list 
INFO >> gamma >> 

Processing services command... 

NAME VALUE 


appliedLastSeqno: 0 


appliedLatency : 21.113 

role : Slave 
serviceName : effectivemysql 
serviceType : local 

started : true 

state : ONLINE 


Finished services command... 


At this time you can delete the files downloaded for Tungsten Replicator, 
as these are now included at TUNGSTEN_HOME/tungsten/tungsten- 
replicator. 


NOTE In this example, Tungsten was used to configure three servers with 
master/slave replication. Tungsten can also be installed in direct mode 
alongside existing MySQL replication, and a simple command can be used to 
take over from native replication on a running system. 


Tungsten Replicator Status Check 
This is a MySQL master/slave topology managed by Tungsten Replicator. 
You can perform a few simple checks on the master: 


$ export PATH-S$HOME/tungsten/tungsten/tungsten-replicator/bin:$PATH 
$ replicator status 

Tungsten Replicator Service is running (PID:5843). 

$ trepctl services 

Processing services command... 

NAME VALUE 


appliedLastSeqno: 0 





appliedLatency : 1.024 

role : master 
serviceName : effectivemysql 
serviceType : local 

started : true 


state : ONLINE 
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Finished services command... 
$ for H in ^echo "alpha beta gamma"^; do echo "*** $H ***"; \ 
trepctl -host $H status | grep applied; done 


At this time we can repeat the replication example from the appendix. 
For example: 


alpha» CREATE SCHEMA IF NOT EXISTS book3; 
alpha» USE book3; 
alpha» SOURCE fill numbers.sqgl 


While only a litmus test, we can confirm nothing obvious is wrong with 
the two slaves by comparing table results: 


$ mysql -utungsten -p -halpha book3 -e "SELECT COUNT(*), SUM(id) FROM numbers" 


$ mysql -utungsten -p -hbeta book3 -e "SELECT COUNT(*), SUM(id) FROM numbers" 


$ mysql -utungsten -p -hgamma book3 -e "SELECT COUNT(*), SUM(id) FROM numbers" 











1048576 549756338176 
+---------- +-------------- + 


Tungsten Replicator Testing 

In order to show replication in various states of operation and verification 
and have a little fun, we can run an additional stored procedure that is a 
little more random and has a much longer execution time: 


alpha> CREATE SCHEMA IF NOT EXISTS book3; 
alpha> USE book3; 
alpha> SOURCE rand fill_numbers.sql 


This will perform a modified version of the simple test case, randomiz- 
ing data that is inserted and performing a large number of iterations: 


$ for H in ^echo "alpha beta gamma" do echo "*** SH **x*"; 
trepctl -host $H status | grep -e applied -e role -e stat; done 

*** alpha *** 

Processing status command... 


applied 
applied 


LastEventId 
LastSeqno 





applied 
role 
state 


Latency 
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mysql-bin.000002:0000000002402714;0 
10638 

0.185 

master 

ONLINE 
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Finished status command... 
*** beta *** 
Processing status command... 


appliedLastEventId mysql-bin.000002:0000000002403150;0 
appliedLastSeqno : 10640 

appliedLatency : 0.859 

role slave 

state ONLINE 


Finished status command... 
*** gamma *** 
Processing status command... 


appliedLastEventId mysql-bin.000002:0000000002403585;0 
appliedLastSeqno : 10642 

appliedLatency r 0.0 

role : slave 

state ONLINE 


Finished status command... 


As you can see, the appl iedLastSeqno shows work is occurring on the 
slaves that are online. The appliedLatency is an indication of slave lag. 
We can delve into the THL to identify the two SQL statements between the 
sequence numbers on the slaves by using the sequence number shown: 


$ thl list -seqno 10641 





SEQH = 10641 / FRAG# = 0 (last frag) 

- TIME = 2012-06-22 12:53:47.0 

- EPOCH# = 0 

- EVENTID = mysql-bin.000002:0000000002403395;0 

- SOURCEID = alpha 

- METADATA = [mysql server id-51;service-effectivemysqgl;shard-book3] 
- TYPE = com.continuent.tungsten.replicator.event.ReplDBMSEvent 

- OPTIONS = [##charset = UTF-8, autocommit = 1, sql_auto_is_null = 0, 
foreign key checks = 1, unique checks = 1, sql_mode = '', 


character set client = 33, collation connection = 33, collation server = 8] 
- SCHEMA - book3 
- SQL(0) = INSERT INTO numbers (id) 
SELECT id + NAME CONST('counter',66148) FROM numbers 
/* | SERVICE = [effectivemysql] */ 


$ thl list -seqno 10642 


SEQ# = 10642 / FRAG# = 0 (last frag) 

- TIME = 2012-06-22 12:53:48.0 

- EPOCH# = 0 

- EVENTID = mysql-bin.000002:0000000002403585;0 

- SOURCEID = alpha 

- METADATA = [mysql server id-51;service-effectivemysqgl;shard-book3] 
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- TYPE - com.continuent.tungsten.replicator.event.ReplDBMSEvent 


- OPTIONS = [##charset = UTF-8, autocommit = 1, sql auto is null = 0, 
foreign key checks = 1, unique checks = 1, sgl mode = '', 

character set client - 33, collation connection - 33, collation server - 8] 

- SCHEMA - book3 

- SQL(0) = DELETE FROM numbers LIMIT 6371 /* | SERVICE = [effectivemysql] */ 


Replication Stoppage Verification 
You can easily stop replication on a slave with: 


$ trepctl -host gamma offline 
The following verifies the stoppage: 


$ trepctl -host gamma status 





NAME VALUE 
appliedLastEventId : NONE 
appliedLastSeqno : -1 
appliedLatency : -1.0 
pendingError : NONE 
pendingErrorCode : NONE 

state : OFFLINE:NORMAL 


And to restart: 


$ trepctl -host gamma online 


Replication Failure Verification 
We can simulate looking into a replication error on a slave with the follow- 
ing destructive command: 


$ mysql -utungsten -p -hgamma book3 -e "DROP TABLE numbers"; 
A review of the Tungsten slave status shows: 


$ trepctl -host gamma status 
Processing status command... 


NAME VALUE 

appliedLastEventlId : NONE 

appliedLastSeqno : 0-1 

appliedLatency : -1.0 

pendingError : Event application failed: seqno-16268 fragno-0 
message-java.sql.SQLException: Statement failed on slave but succeeded on master 
pendingErrorCode : NONE 

pendingErrorEventId : mysql-bin.000002:0000000003609171;0 


pendingErrorSeqno : 16268 
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pendingExceptionMessage: 
succeeded on master 


[effectivemysql] */ 
state 
timeInStateSeconds 


uptimeSeconds 
Finished status command. 


java.sql.SQLException: Statement failed on slave but 


TRUNCATE TABLE numbers /* SERVICE = 


: OFFLINE: ERROR 
: 43.215 


168578.375 


As you can see, the state indicates an error and the pendingError 
related columns provide details of the failure. Following correction of the 
situation causing the error, you can restart the slave and verify the new 


state situation: 


$ mysql -utungsten -p -hgamma book3 -e "CREATE TABLE numbers(id INT NOT NULL) ;" 
$ trepctl -host gamma online 


$ for H in "echo "gamma"^ 


; do trepctl -host $H status | \ 


grep -e applied -e role -e stat; done 


Processing status command... 


appliedLastEventId 
appliedLastSeqno 
appliedLatency 
role 

state 

Finished status command. 


: OFFLINE: ERROR 


Processing status command... 


appliedLastEventId 
appliedLastSeqno 
appliedLatency 
role 

state 


: mysql-bin.000002:0000000003620196;0 


16319 
0.0 


: Slave 
: ONLINE 


Replication Failover 
The following steps are used to perform a failover in the example Tungsten 
Replicator environment: 


e Confirm an operational cluster. 


e Simulate a master failure. 


Verify the remaining cluster status. 


e Select a new master. 


Fail over to new master. 


Verify cluster operations. 
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Verify Cluster First, verify the state of the current master/slave cluster: 


$ trepctl -host alpha heartbeat 
$ for NODE in alpha beta gamma 
do 
echo "#${NODE}" 
trepctl -host ${NODE} services | simple services 
done 














#alpha 

effectivemysql [master] 

seqno: 2 - latency: 0.101 - ONLINE 
#beta 

effectivemysql [slave] 

seqno: 2 - latency: 0.625 - ONLINE 
#gamma 

effectivemysql [slave] 

seqno: 2 = latency: 0.994 - ONLINE 


Simulate a Master Failure The next step is to simulate some load on the 
master server with: 


alpha» SOURCE rand fill numbers.sql 


You can simulate a master server failure by taking the master server 
offline: 


$ trepctl -host alpha offline 


Verify Cluster The next step is to confirm replication on the attached 
slaves is up to date based on the transaction logs stored on the slaves: 


$ for NODE in beta gamma 

do 
MAXSTORED-^trepctl -host $NODE status | grep maximumStoredSeqNo | \ 
awk '{print $3]'^ 
trepctl -host $NODE wait -applied $MAXSTORED 

done 


Identify New Master Check the status of the remaining slave nodes to 
identify the most up-to-date server: 


$ for NODE in beta gamma 
do 

echo "#SNODE" 

trepctl -host $NODE services | simple services 
done 
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#beta 

effectivemysql [slave] 

seqno: 620 - latency: 33.582 - GOING-ONLINE: SYNCHRONIZING 
#gamma 

effectivemysql [slave] 

seqno: 650 - latency: 25.289 - GOING-ONLINE: SYNCHRONIZING 


From this output we can determine that the gamma server has the most 
current transaction sequence number and is the best candidate for the new 
master. 


Perform Master Failover The master failover involves isolating the 
cluster nodes from operations, setting the role of the new master, and 
defining the new master for any slaves: 


trepctl -host beta offline 

trepctl -host gamma offline 

trepctl -host gamma setrole -role master 

trepctl -host gamma online 

trepctl -host gamma services 

trepctl -host beta setrole -role slave -uri thl://gamma 
trepctl -host beta online 


VoU Ur Ur Ur UV UY 


Verify Operational Cluster The final step of the failover process is to 
verify the state of the new cluster: 


$ trepctl -host $node3 heartbeat 
$ for NODE in beta gamma 
do 
echo "#SNODE" 
trepctl -host $NODE services | simple services 
done 





#beta 

effectivemysql [slave] 

seqno: 651 - latency: 0.969 - ONLINE 
#gamma 

effectivemysql [master] 

seqno: 651 - latency: 0.427 - ONLINE 


Recommended Configuration 

Tungsten is a Java process, so understanding and managing the Java Virtual 
Machine (JVM) memory usage is important for optimal performance. Be 
sure to review the appropriate documentation and monitor JVM memory in 
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your environment. It is recommended the JVM be configured with 1GB of 
RAM. The following are recommended THL settings: 


€ bufferSize=128KB This is used for reads and writes from/to storage. 
This should never be less than the size of pages in persistent storage. 


€ doCheckSum-true This enables checksums on records. This can 
have an impact on log performance but allows unambiguous 
detection of log record corruption. 


e fsyncOnFlush-true This performs a true fsync on disk write. 
This can be slow on storage with no battery backed write cache 
(BBWC) but required for crash-safe slaves. 


€ logFileSize-100B This is the maximum number of bytes to 
write before rotating to a new log file. 


e Ensure at least 1GB in page cache. 
e Use innodb flush method-O DIRECT if onboard with MySQL. 


e You should have 500M to 2GB free memory on the system for the 
OS page cache to improve parallel replication. 


Alternative Tungsten Deployments 


Tungsten Replicator provides for a variety of replication topologies that are 
not possible with native MySQL replication. One of these is the ability for 
a slave to have multiple masters, also referred to as fan-in replication. 


Fan-In Replication 
Fan-in replication, as shown in Figure 6-1, allows a MySQL instance to 
receive replication requests from multiple masters. When combined with 
appropriate filtering, this configuration can provide a centralized data ware- 
house of distribution data with no additional data manipulation or the com- 
mon Extract, Transform, Load (ETL) step. With the advanced capabilities of 
Tungsten Replicator supporting different RDBMS and NoSQL products, it is 
possible for the fan-in instance to support data from varying sources. 
Tungsten Sandbox includes a fan-in example that can be configured 
with a single command. The presentation at http://www.percona.com/live/ 
mysql-conference-2012/sessions/build-simple-and-complex-replication- 
clusters-tungsten-replicator and blog post at http://mysql-replication- 
blog.blogspot.com/2011/12/testing-tungsten-fan-in-replication.html 
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Master 1 Master 2 Master 3 








Fan in Slave 


Figure 6-1 Fan-in replication 


provide additional references for this type of deployment. The following 
script provides a skeleton example. 


#!/bin/bash 


MASTER HOSTS- (alpha beta) 

FAN IN HOST-gamma 

FAN IN DATASOURCE-$(FAN IN HOST] 

SERVICES- (alpha beta gamma) 

SLAVE SERVICES- (alpha beta) 

UNGSTEN BASE-$HOME/installs/fan in 

REPCTL-$TUNGSTEN BASE/tungsten/tungsten-replicator/bin/trepctl 

HH 

H4 First, we create a master service in each server 

HH 

N=0 

for HOST in ${MASTER_HOSTS[*]} $(FAN IN HOST} 

do 

./tools/tungsten-installer \ 
--master-slave \ 
--master-host=${HOST} ^ 
--cluster-hosts=${HOST} \ 
--datasource-port=3306 \ 
--datasource-user=tungsten \ 
--datasource-password=secret \ 
--home-directory=${TUNGSTEN BASE} \ 
--datasource-log-directory=/var/lib/mysql \ 
--datasource-mysql-conf=/etc/my.cnf \ 
--service-name=$ {SERVICES [$N] } \ 
--start 
N-$ ( ($N+1) ) 

done 




















HH 

## After the remote masters are created, 

## we create the corresponding slave services in the fan-in host 
HH 

N=0 
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for HOST in ${MASTER_HOSTS [*] } 

do 

STUNGSTEN BASE/tungsten/tools/configure-service \ 
-c \ 
--quiet \ 
--host=${FAN IN HOST} ^ 
--datasource=${FAN IN DATASOURCE} \ 
--role=slave \ 
--service-type=remote \ 
--master-thl-host=${HOST} \ 
--sve-start \ 
--skip-validation-check=THLStorageCheck \ 
--local-service-name=delta \ 
--log-slave-updates=true \ 
${SERVICES [$N] } 
N-$ ( ($N+1) ) 

done 





## 
## Next, we test replication, 

## by creating a different schema in each remote 

## master, and collecting the results from the fan-in slave 
## 
N=0 
for HOST in ${MASTER_HOSTS [*] } 
do 





mysql -h${HOST} -utungsten -psecret -e \ 
"DROP SCHEMA IF EXISTS ${SERVICES[SN]};" \ 
"CREATE SCHEMA ${SERVICES[SN]};" ^ 

"USE ${SERVICES[SN]};" \ 

"CREATE TABLE test S(SERVICES[SN]]) (i int)" \ 
"INSERT INTO test_${SERVICES[$N]} VALUES ($N)" 
N=S ( (SN+1) ) 











done 
sleep 3 


for SCHEMA in ${SLAVE_ SERVICES [*] } 


do 
mysql -v -h$(FAN IN HOST} -utungsten -psecret ${SCHEMA} \ 
-e "SELECT * FROM test $SCHEMA" 

done 

HH 


## Finally, we show the replication summary using trepctl 


HF 


for HOST in ${MASTER_HOSTS[*]} $(FAN IN HOST} 
do 
STREPCTL -host $HOST services | simple services 
done 
exit 0 
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The following output is produced when run. 


prcne + 
Li || 
qenesus + 
| o | 
posses + 


alpha [master] 


seqno: 8 - latency: 


beta [master] 


seqno: 8 - latency: 


alpha [slave] 


seqno: 8 - latency: 


beta [slave] 


seqno: 8 - latency: 


delta [master] 





seqno: 90 - latency: 


Direct Replication Mode 


ONI 


ONI 


ONI 


ONI 


ONI 


LINE 


LINE 


LINE 


LINE 





LINE 
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Tungsten Replicator can be used in conjunction with MySQL replication in 
a production environment. The following script shows how a MySQL mas- 
ter/slave replication topology can use native MySQL replication, then have 
Tungsten Replicator take over for a period of time to reduce lag with paral- 
lel replication capabilities, and revert back to native MySQL replication. 


#!/bin/sh 


TUNGSTEN_BASE=$HOME/installs/direct 


[ ! -d $TUNGSTEN_BASE ] && mkdir -p $TUNGSTEN BASE 


HOW MANY CHANNELS-$1 


[ -z $HOW MANY CHANNELS ] && HOW MANY CHANNELS-5 


TREPCTL-$(TUNGSTEN BASE]/tungsten/tungsten-replicator/bin/trepctl 


MASTER-"alpha" 
SLAVE1="beta" 
SLAVE2-"gamma" 
NUM SLAVES-"2" 
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Li 
## Installing native MySQL replication on the slaves 
HH 
ist 
while [ $i -le $(NUM SLAVES) 
do 
SLAVE="slave${i}" 
mysql -h${SLAVE} -e "STOP SLAVE; " \ 
"CHANGE MASTER TO MASTER HOST-'S(MASTER]'," V 
"MASTER PORT-3306, MASTER USER-'tungsten', MASTER PASSWORD-'secret'; " \ 


"START SLAVE" 
i-^expr $i + 1^ 
done 


Li 

## To test a direct slave with parallel replication, you should: 

## * create several database schemas 

## * start several dozen threads that update those databases concurrently 
## * stop the slaves for a few minutes, to accumulate some lag 

## * restart the slaves 

## * finally, let Tungsten Replicator take over on slave, with the command 


./tools/tungsten-installer \ 
--direct \ 
--master-host=${MASTER} \ 
--slave-host=${SLAVE2} \ 
--master-user-tungsten V 
--master-mysql-conf-/etc/my.cnf \ 
--Slave-mysql-conf-/etc/my.cnf \ 
--slave-user=tungsten V 
--master-password-secret \ 
--Slave-password-secret V 
--service-name=effectivemysql \ 
--channels=${HOW MANY CHANNELS} \ 
--home-directory=${TUNGSTEN BASE} \ 
--buffer-size-100 \ 
--native-slave-takeover \ 
--start-and-report 


STREPCTL status 
STREPCTL status -name shards 


STREPCTL status -name stores 


mysql -h ${SLAVE2} -e 'SELECT * FROM tungsten effectivemysql.trep commit seqgno' 


To hand over replication back to MySQL native replication, 





do the following: 
* check 'SHOW SLAVE STATUS\G' Should be stopped at the point where Tungsten 
took over 
* put the replicator offline (trepctl offline) 
* check 'SHOW SLAVE STATUS\G' again. Should be updated to the latest 








Hho db dt db dt out 
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#H position used by Tungsten 
HH * run 'START SLAVE', and the native replication will resume. 
#H 


## If you then put the replicator back online, it will take over again. 


For more information see http://code.google.com/p/tungsten-replicator/ 
wiki/TRCBasicInstallation£Taking over replication from. a MySOL 
slave in direct mode. 


Unique Characteristics 


You may not consider replication to be a disruptive technology; however, 
features of replication that can influence and support the changes occur- 
ring in the database and data store space, especially with managing larger 
and varying amounts of data, are disruptive. 

Tungsten has some characteristics that set it apart uniquely from both 
MySQL and other third-party vendors, versions, flavors, patches, add- 
ons, etc. 


1. Tungsten can replicate data to and from different disparate data 
sources, including Oracle, MongoDB, Postgres, and Vertica. That 
list, I am sure, will grow. 


2. Tungsten runs on stock standard MySQL; no installation of modified 
MySQL necessary, no change to running software, installation 
procedures, monitoring, etc. Tungsten is an additional product that 
simply extends MySQL. Tungsten, of course, requires installation, 
management, monitoring, etc. 


3. Tungsten allows for the fan in architecture. That is where a slave 
server can receive and manage information from multiple masters, 
including different products. This has just simplified the data 
warehouse process to a software installation and configuration 
process; no additional transformation or export/import necessary. 


4. Tungsten Replicator can be installed in an existing MySOL 
replication environment and can be used to take over native 
replication with a simple command. 
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Continuent Tungsten 


The commercial offering of Tungsten Replicator is called Continuent Tung- 
sten. Some of the additional features in the enterprise version include 


€ Advanced installation for regular master/slave and multisite 
topologies. 


Database virtualization. A Tungsten cluster is seen and administered 
as a single database. 


Automatic failover (or manual failover with a single command when 
automatic is disabled). 


Automatic and manual recovery of failed slaves. 


Centralized management tool that includes backup and restore 
automation. 


Single command master switch with zero downtime. 


Transparent routing and connectivity that allow applications to see 
the cluster as a single server. No virtual IPs are necessary, even for 
cross-site switches. 


Continuent also provides commercial 24/7 support for Tungsten.You can 
view a feature matrix at http://continuent.com/solutions/featurematrix. 


Continuent Wrap-Up 


This section only scratches the surface of the features and functionality of 
Tungsten Replicator. Indeed this product could easily have a full book for 
readers to understand and appreciate the depth. These examples are in- 
cluded to teach you to walk before learning to run with Tungsten. One of the 
true benefits and complexities of Tungsten is the amount of custom options 
for huge environments that replicate efficiently across global locations, pro- 
viding parallelizing data flow and hot failovers of entire Tungsten clusters 
seamlessly. These features are available for a startup to the largest web 
properties around. 


Get More Help 
You can join the discussion group, and view and log any issues at http:// 
tungsten-replicator.org. 
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SchoonerSOL 


SchoonerSQL by Schooner Information Technology (recently acquired by 
SanDisk) provides a commercial synchronous MySQL replication solution 
using a modified version of InnoDB. The documented features include no 
loss of data, instance failover, and automated recovery capabilities. A 
SchoonerSQL solution can also include traditional MySQL replication for 
additional slaves. As a commerical product, there is no access to the soft- 
ware without a formal pre-sales process. A request to review the software 
for this book was not granted. 


MySQL Replication Listener 


The MySQL Replication Listener, available from https://launchpad.net/ 
mysql-replication-listener, is an open source C++ library that can process 
a replication stream. This is a MySQL binary log API for capturing any 
data changes in MySQL. This can be used to read and decode information, 
and then custom code that can apply the data manipulation based on 
need, for example, populating a dedicated full text tool or synchronizing 
with a caching system. This tool can also be used to read and analyze the 
binary log. 

This is a simple and extensible API that can use the network transport 
(i.e., a master server) or a file transport (i.e., a binary log file) to read and 
decode the replication events. Dr. Lars Thalmann and Dr. Mats Kindahl, 
two key members of the Oracle/MySQL replication team and co-authors of 
MySQL High Availability (O’Reilly, 2010), provide a more detailed explana- 
tion in the presentation “Binary Log API: A Library for Change Data Cap- 
ture Using MySQL,” obtained at http://www.oscon.com/oscon2011/public/ 
schedule/detail/18785. A detailed example of how to use the API can be 
found at http://intuitive-search.blogspot.se/2011/07/binary-log-api-and- 
replication-listener.html. 


MySQL in the Cloud 


Two existing cloud product offerings extend traditional MySQL replication 
with synchronous replication options. 
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Amazon RDS for MySOL 


Amazon RDS for MySQL provides for read replicas, which are traditional 
MySQL slaves using replication. RDS also provides a multi-AZ deployment 
where proprietary synchronous replication is used to provide a standby 
replica. Amazon RDS is also documented to automatically failover; however, 
this has not been tested for confirmation. While a read replica is available for 
general read access, a multi-AZ replica is not accessible before a failover. 
The Effective MySQL: Backup and Recovery book (McGraw-Hill, 2012) has a 
detailed section on the features and use of Amazon RDS for MySOL. 
Amazon RDS for MySQL is an established product offering with several 
years of general availability. More information is available at http://aws 
.amazon.com/rds/mysgql/. 


Google Cloud SOL 


The Google cloud offering provides a proprietary synchronous replication 
configuration by default. There is no traditional asynchronous option. 
Google Cloud SOL does not provide any access to the synchronous copy 
and will also automatically manage failover without any need for human 
intervention. The Effective MySQL: Backup and Recovery book has a detailed 
section on the features and use of Google Cloud SOL. More information is 
available at https://developers.google.com/cloud-sql/. 


Other Offerings 


The MySQL ecosystem has included its share of new product offerings that 
pop up and claim to solve MySQL replication and scale out issues with 
various features. Many simply simulate the MySOL protocol, i.e., the com- 
munication that MySQL uses between client connectors and the MySQL 
kernel. This book is specifically designed for MySOL replication that is part 
of the core (and well established and stable) MySOL product. Reading 
about the problems these products are trying to solve is applicable when 
discussing replication. A few include (in alphabetical order) Clustrix (http:// 
www.clustrix.com/), ScaleARC (http://www.scalearc.com/), ScaleDB (http:// 
www.scaledb.com/), and Xeround (http://xeround.com/). No evaluation has 
been made for comparison with MySOL and what is stated at these respec- 
tive company websites. 


Extending Replication for Practical Needs 205 


Conclusion 


MySQL replication is a core component for designing scale out replication 
architectures. In this chapter we have discussed several commercial strength 
products and other features that extend traditional MySQL replication with 
many enterprise class features. As with many options, appropriate testing 
and verification in your unique environment are important to make an 
informed decision on what is most applicable in any given situation. 
Examples and links in this chapter are available for download from 
http://EffectiveMySQL.com/book/replication-techniques. 
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MySQL Configuration 
Options 


MySQL 5.6 now supports almost 400' configurable system variables. A 
number of these variables have a direct effect on how MySQL will operate 
when using replication. Understanding what system variables do and how 
they change the behavior of the MySQL server will help ensure that MySQL 
replication operates as expected. 


* 391 in 5.6.5-m8, 319 in 5.5.24, 276 in 5.1.63, and just 239 in 5.0.95 for master servers. The count 
also changes depending on compile and startup options and if replication or plugins are 
enabled. 
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In this chapter we will discuss 
e Binary logging variables 
e Replication system variables 
e Replication security variables 
e New MySQL 5.6 variables 


e Replication-specific grants, commands, and functions 


About MySOL System Variables 


Unless otherwise specified, all variables are available in MySQL 5.5, the 
current GA version. While every attempt has been made to document im- 
portant and relevant MySQL 5.6 configuration variables, as a development 
release of MySQL these are subject to change. Always refer to the MySOL 
5.6 documentation for current information at http://dev.mysql.com/doc/ 
refman/5.6/en/server-system-variables.html. 

This chapter does not discuss any variables that are part of any third-party 
products or MySQL variants mentioned during this book. The applicable 
sections describe references to the respective product documentation. 

The MySQL reference manual is the best resource of information available. 
All replication-specific variables can be found at http://dev.mysql.com/ 
doc/refman/5.5/en/replication-options-table.html. 


Binary Logging 


These initial variables are required settings for the configuration of 
MySQL binary logging, and are essential components when using MySQL 
replication: 


e server id This mandatory variable is a unique number for the 
server within the current MySQL topology. 


€ log bin This enables the binary log and is mandatory for 
replication on the master host. This variable also defines the 
basename of the binary log files. The default file path is the data 
directory with a filename of the hostname +‘-bin’. 
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€ log bin index This variable defines the name of the index that 
holds a list of all current binary logs. This will default to the 1og bin 
basename with an . index extension. 


€ binlog format This variable controls the type of binary logging. 
The value of STATEMENT, the default, logs the actual SOL statement 
to the binary log. The value of ROW will log changed data blocks to 
the binary log. The value of MIXED will choose the most applicable 
method for the given statement necessary to ensure data consistency. 
This option can be specified on a per session level, for example, with 
SET SESSION.binlog format - ROW. 


€ binlog row image (5.6) This variable alters the amount of data 
block information that is used when binlog format is ROW.The 
value of £u11, the default, logs full before and after row images. The 
values of minimal and blob alter the columns used in the before 
and after images to reduce disk space and network bandwidth. 


e binlog do db & binlog ignore db These variables on the 
master host limit which statements are logged to the binary log 
based on the specified database name, preceded by a USE qualifier. 
Multiple database values can be specified using multiple lines in the 
my . cnf. For example: 

#my. cnf 

[mysqld] 

binlog_do_db = book3 

binlog_do_db = mysql 

CAUTION Use of báinlog do dband binlog_ingnore_db can make a 

binary log unusable in a point in time recovery of a full primary database. 
These options are also incomplete, as they require all SQL to be preceded by an 
applicable USE, and do not handle cross-schema joins as you would expect. 


€ binlog cache size This cache is used to hold changes that are 
to be written to the binary log during a transaction. Increasing this 
value for very large transactions can possibly increase performance. 


€ binlog stmt cache size This variable specifies the size of the 





cache for the binary log to hold non-transactional statements during 
transactions on a per client basis. There may be a benefit to 
increasing this value using large non-transactional statements. 
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e binlog row event max size This variable represents the 





maximum size of a row-based binary log event. 


e max binlog size This is the maximum size of the binary log file 
before a new file is created. This defaults to and can be a maximum 
of 1GB. It is possible the resulting file may be larger, as binary logs 
contain completed transactions. The FLUSH BINARY LOGS 
command will also dynamically close the current binary log and 
create a new file. 


€ expire logs days This variable defines the number of days 
binary log files are retained. Files older than the number of days are 
removed (similar in operation to a PURGE MASTER LOGS 
command) when a new binary log file is created. A check is 
performed to confirm no connected slave server is currently using a 
binary log that is being purged. There is no check for any slaves that 
may not be connected. 


e sync binlog This variable defines when the binary log file is 
physically synced to disk. The value is the number of statements 
executed. The best durability is a value of 1, meaning at the worst case 
only one transaction could be lost in a system failure with appropriate 
redundancies. The default is 0, which delegates to the operating 
system to manage the sync. With an applicable RAID controller card 
and Battery Backed Write Cache (BBWC), this hardware feature may 
provide the same level of durability as a value of 1. 


e log bin basename (5.6) This new variable defines the complete 
path to the binary log. In previous versions the log bin variable was 
used to define the enabling of the binary log (e.g., ON) and an 
optional basename in the configuration. This basename was never 
visible with SHOW GLOBAL VARIABLES. The following shows the 
log-bin variable defined in MySQL Sandbox for versions 5.5 and 5.6, 
and shows the corresponding values represented by SHOW GLOBAL 
VARIABLES to demonstrate the new 1og bin basename variable. 


For MySQL 5.5: 


$ cd $HOME/sandboxes/rsandbox 5 5 24 

$ grep log-bin master/my.sandbox.cnf 
log-bin=mysql-bin 

mysql55> SHOW GLOBAL VARIABLES LIKE  'log bin%'\G 
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ckckokckckckokck ck okokck ck ckokck ck ckckck ck ck kc kk 1], POW CKCkCkckckckck ck kk k kk kokck ck k k k ck ck ok ck ck ke k 


Variable name: log bin 
Value: ON 
kckckckckckckckckckckckckckckckckck ck ck ck ck k ck k kk 2., row kk kckckckckckckckckckckckckckckck kc kckck ko ko ko 


Variable name: log bin trust function creators 
Value: OFF 


For MySQL 5.6 


$ cd SHOME/sandboxes/rsandbox 5 6 5 
$ grep log-bin master/my.sandbox.cnf 





log-bin=mysql-bin 


Note that the configuration is identical to MySQL 5.5; however, 
there is now the 1og bin basename variable: 


mysql56> SHOW GLOBAL VARIABLES LIKE 'log bin$' VG 
kckckckckckckckckckckckckckckckckck kc k kc k k kk kk 1. row kk kckckckckckckckckckckckckckckckckck kk ko 
Variable name: log bin 

Value: ON 
ckckckckckckckckckckckckckck ckckckckck kck k ck k kk 2., row KEK KKK KKK KKK KKK KKK KKK KKK ko ko 
Variable name: log bin basename 

Value: /home/user/sandboxes/rsandbox 5 6 5/master/data/mysql-bin 
kckckckckckckckckckckckckckckckckck ck ck ckckck ck k kk 3., row  Ckckckckckckckckckckckckckckckckckckck kc kckck ck ko ko 
Variable name: log bin index 

Value: /home/user/sandboxes/rsandbox 5 6 5/master/data/mysql-bin. index 
kckckckckckckckckckckckckckck ck ckck ck ck ck ck k k k kk 4., row KEK kckckckckckckckckckckckckckckckckckckck ko ko ko 
Variable name: log bin trust function creators 

Value: OFF 





e binlog rows query log events(5.6) This new variable is used 





to provide additional row-based logging information with the binary 
log and can be used for debugging purposes. One example is the 
representation of the actual SOL statement when using row-based 
replication: 


master» CREATE SCHEMA IF NOT EXISTS book3; 

master» USE book3; 

master» DROP TABLE IF EXISTS bl events; 

master» CREATE TABLE bl events (ID INT NOT NULL); 
master» SET SESSION binlog format-ROW; 

master» FLUSH BINARY LOGS; 

master» SHOW BINARY LOGS; 

master» SET SESSION binlog rows query log events-ON; 
master» INSERT INTO bl events VALUES (1), (2), (3); 
master» SET SESSION binlog rows query log events-OFF; 
master» INSERT INTO bl events VALUES (1), (2),(3); 
master» SHOW BINLOG EVENTS IN 'mysql-bin.000009'; 





| 4 | Format desc |Server ver: 5.6.5-m8-log, Binlog ver: 4 
| 120 | Previous gtids |B9A8FDE1-B4A3-11E1-921D-B499BAF75D68:1-9 
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187 Gtid SET @@SESSION.GTID_NEXT= 'B9A8FDE1-B4A3-11E1- 
231 Query BEGIN 
300 Rows query # INSERT INTO bl events VALUES (1), (2), (3) 
359 Table map table id: 72 (book3.bl events) 
408 Write rows table id: 72 flags: STMT END F 
452 Xid COMMIT /* xid=75 */ 
479 Gtid SET @@SESSION.GTID_NEXT= 'B9A8FDE1-B4A3-11E1- 
523 Query BEGIN 
592 Table_map table_id: 72 (book3.bl_events) 
641 Write rows table id: 72 flags: STMT END F 
685 Xid COMMIT /* xid=77 */ 
R----- ss Pie Sei qorzze£lescckceruesecuecgusc eec 


The previous example also shows the GTID-specific binary log entries. 
If gcid mode is not enabled, this output will not be displayed. The full 
list of columns for SHOW BINLOG EVENTS can be seen with \G. For 
example: 


master» SHOW BINLOG EVENTS IN 'mysql-bin.000009'\G 
kckckckckckckckckck ck ckckckck ck ckck kc k kk k kk kk 1. row kk KKK RK KEKE KKK KKK KKK ko ko ko 
Log name: mysql-bin.000009 
Pos: 4 
Event type: Format desc 
Server id: 1 
End log pos: 120 
Info: Server ver: 5.6.5-m8-log, Binlog ver: 4 


e binlog order commits (5.6) This group commit variable 
determines the order of committed transactions to support parallel 
operations. 





€ binlog max flush queue time (5.6) Thisgroup commit 
variable determines how many milliseconds to keep reading 
transactions from the flush queue before proceeding. 


€ binlog flush log at timeout (5.6) This variable determines 
when to flush the binary log every N seconds. 





This is nota complete list of possible variables for binary logs. See http:// 
dev.mysql.com/doc/refman/5.6/en/replication-options.html for detailed 
information of these replication options. 


MySQL Replication 


These variables affect the way MySQL replication behaves. Whether a slave 
host is set to only replicate certain databases, skip certain errors, and/or is 
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setup in a unique chain topology, itis important to know how the following 
will affect your setup: 


e relay log The relay logs hold replicated database changes 
retrieved from the master binary log and written with the I/O thread. 
If not specified, this file path will default to the MySQL data directory, 
the server hostname, and the MySQL port. 


e relay log index This variable defines the name of the relay log 
index that holds the names of all the relay logs available. The default 
filename is the relay 1og variable value with the extension . index. 


e replicate do db & replicate ignore db These variables 
are used to filter which recorded master binary log statements are 
applied on the slave. Their use is much like bàinlog do db and 
binlog ignore db options on the master host. For multiple 
database values, specify the options multiple times. There are similar 
replicate- options for tables and for wildcard database/table 
matching. 


CAUTION The replicate do dband replicate ingnore dbcan 
cause errors, as they require all SQL to be preceded by an applicable USE and 
do not handle cross-schema joins as you would expect. 


e slave skip errors Replication error codes can be skipped 
automatically when specified with this variable. Normally, replication 
will stop when the SOL thread encounters an error; however, this 
variable will cause the SOL thread to skip those errors listed in the 
variable value. It is rarely a good idea to specify a value for s1ave _ 
Skip errors, because there is no accountability of the occurrences 
of these silent errors, which will generally lead to data drift and/or 
loss of data integrity. The format of the value for this variable is a 
comma separated list of MySOL error numbers. 


€ slave exec mode There are two valid values for s1ave exec | 
mode, IDEMPOTENT and STRICT. This variable is used for replication 
conflict resolution and error checking. If the value is set to 
IDEMPOTENT (default for NDB), the slave will not error out during 
duplicate key or no key found errors. The IDEMPOTENT value is 
useful with a system that is set up in a multi-master or circular 
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replication fashion. When the value is set to STRICT, the default, 
replication will stop on duplicate key and no key found errors. 


€ log slave updates When defined and binary logging is 
enabled on a slave, all replicated changes from the SOL thread are 
also written to the slave server binary log. This option is used to 
chain multiple nodes together through replication. For example, if 
you have three servers (A, B, and C) and want to connect them in a 
chain you would use log slave updates on B. B would replicate 
from A, and C from B, forming a chain, (A -> B -> C). 


€ relay log purge This variable controls how the relay log files 
are purged. The default of 1 specifies that the relay log files are 
removed when they are no longer needed for applying replication 
events. A value of 0 retains the log files. 


e read only This variable defines that the slave will not accept 
DML or DDL statements other than those applied by the replication 
slave SOL thread. The exception is a user with SUPER privilege will 
override this setting. 


e skip slave start By default, when a slave server starts, an 
implied SLAVE START occurs. With this variable specified, the slave 
is not automatically started and must be performed manually with 
START SLAVE. 


€ sync relay log,sync relay log info These variables 
control how frequently a file sync is performed on the respective 
relay log and relay log info file. The number represents the name of 
executed SOL statements to apply before action. The default is 0; the 
safest durability setting is 1. 


e report host This optional variable provides a string for the slave 
that is reported with SHOW SLAVE HOSTS on the master. 


e slave-max-allowed-packet (5.6) This new variable defines the 
maximum allowed packet size for the slave SOL and I/O threads. 


e relay-log-recovery (5.60 This new variable, when enabled 
(disabled by default), will discard any unprocessed slave relay log 
events that have not been applied and will initiate obtaining the 
events from the master. 
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This is not an exhaustive list of all replication-related variables. Full 
details can be found at http://dev.mysql.com/doc/refman/5.6/en/replication- 
options-slave.html. 


Semisynchronous Replication 


A new feature in MySQL 5.5 is the ability to define semisynchronous repli- 
cation. This is an improvement on classic asynchronous replication, as the 
master server waits for a confirmation from a slave that the event has been 
received and recorded on a different server before returning a success indi- 
cator to the calling client. The following semisynchronous variables will only 
be visible following the installation of the necessary plugins as described in 
Chapter 3. These variables control master/slave semisynchronous operation. 
On the master: 


e rpl semi sync master enabled When set to ON, semisyn- 
chronous replication may be initiated by the master. At least one 
slave must also have the corresponding variable set to ON for 
semisynchronous replication to become operational. The SHOW 
GLOBAL VARIABLES LIKE ‘rpl_semi%’ can be used to confirm and 
monitor semisynchronous replication. 


€ rpl semi sync master timeout The master will wait a default 
of 10,000 milliseconds for a response from any slave configured to 
use semisynchronous replication before the master will revert to 
asynchronous replication and return a response to the client. This 
can be set to a lower value if applicable. 

e rpl semi sync master wait no slave This value controls 
how the master will wait for a timeout from one or more slaves before 
reverting to asynchronous replication. The default value is ON. 

€ rpl semi sync master trace level This defines the level of 
debugging logging. The allowed values are 1 (general level logging), 
16 (detailed level logging), 32 (network wait logging), and 64 
(function level logging). 


On the slave: 


e rpl semi sync slave enabled When set to ON, 





semisynchronous replication on the slave is possible. 
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e rpl semi sync slave trace level This defines the level of 
debugging logging. The allowed values are 1, 16, 32, and 64. 





NOTE MySQL will disable semisynchronous replication and revert 
automatically to asynchronous replication when any error occurs or slow 
network overhead exceeds the timeout, so slave servers can continue to operate. 
This may lessen your high availability requirements. Adequate monitoring of 
MySQL status variables and the MySQL error log is very important to 
determine this situation and rectify accordingly. 


Refer to Chapter 3 for the use and demonstration of these variables in 
conjunction with loading the necessary plugins. More information can also 
be found in the MySQL Reference Manual at http://dev.mysql.com/doc/ 
refman/5.5/en/replication-semisync.html. 


Security 


Support for SSL communication with MySQL has been around since ver- 
sion 4.0.0 (October 2001). In a recent survey of over 1000 people at the 2012 
Percona Live MySQL conference keynote, less than 1 percent indicated using 
SSL. The need to use SSL is more prevalent today with the use of cloud 
services. To improve the future adoption of improved security, starting 
with MySQL5.6, starting a slave without SSL will produce a warning. These 
variables define SSL usage for securing client/server and replication 
stream communications: 


e ssl This variable states that the MySQL server permits SSL 
connections. This option does not state that connections require SSL. 
See the GRANT command described later. 


e ssl-ca This variable is used to identify the Certificate Authority 
(CA) certificate file. 


e ssl-cert This identifies the server public key file. This is used in 
client authentication with the CA certificate. 


e ssl-key This identifies the server private key file that is used for 


confirmation of provided security credentials from the client. 


TIP Ensure that you also adequately secure on the file system appropriate access 
to the SSL certificate files defined with these options. 
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The GRANT command is used to ensure user connections requiring se- 
cure communication using SSL are defined with the REQUIRE SSL syntax. 
This syntax also enables additional SSL attributes, including X509, ISSUER, 
SUBJECT, and CIPHER, to further limit SSL authorization. See http://dev 
-mysql.com/doc/refman/5.5/en/grant.html for the full range of syntax 
options. 

Chapter 3 provides a detailed example of the setup and use of SSL in 
MySQL. 

Related SSL variables not described here include skip-ssl, ssl- 
capath, ssl-cipher, and ssl-verify-server-cert. For more infor- 
mation see http://dev.mysql.com/doc/refman/5.5/en/secure-basics.html. 


MySQL Server Variables 


The MySQL server has a number of general variables that can affect 
MySQL replication or are recommended with certain tools that are associ- 
ated with MySQL replication: 


€ have dynamic loading This variable defines if the dynamic 
execution of plugins is supported, for example, when using the 
semisynchronous plugin. 


è auto increment increment This variable defines the increment 
value that is used for an AUTO INCREMENT column in a table. The 
default value is 1. This is applicable in a multi-master environment 
when it is beneficial to change (for example, to 2). When combined 
with auto increment offset,this can ensure no possible collision 
detection for an auto increment primary key if writing to multiple 
servers. This is a global setting for all tables in a given instance. More 
information can be found in Chapter 4. 


e auto increment offset This variable defines the starting value 
of an AUTO INCREMENT column for a table. The default value is 1. 
As described with auto increment increment, this variable is 
used in multi-master environments to manage uniqueness in 
MySQL topology of auto incrementing primary key value. 


e default storage engine This variable defines the storage 
engine that is used when not specified with the CREATE TABLE 
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statement. The default (from MySQL 5.5) and recommended value is 
InnoDB, which is a transactional storage engine. The historical value 
is MyISAM. For more information about MySQL storage engines, 
see http://dev.mysql.com/doc/refman/5.5/en/storage-engines.html. 


e max allowed packet This defines the maximum size of a 
communication packet of MySOL information that can be sent from 
a client to a MySQL server. 


e bind address By default, MySQL will accept network TCP/IP 
communication on the defined port for all system IP addresses. Use 
bind address to limit communications to an individual IP address. 
When set to localhost, or the loopback address (e.g., 127.0.0.1), 
communication to the database is only possible on the server. This 
option accepts any IPv4 or IPv6 address. 


InnoDB Variables 


e innodb flush log at trx commit This variable defines the 





level of durability for InnoDB log transactions when they are written. 
The default value of 1 will write and flush every log transaction to 
disk. This is the safest method for durability. A common setting 
based on business needs and an appropriate RAID controller is a 
value of 2. This writes all transactions to disk; however, it only flushes 
to disk approximately once per second. The final permissible value is 
0, which will only write and flush approximately once per second. 


e innodb locks unsafe for binlog This variable controls how 
InnoDB manages row level locking for operations on a range of rows. 
The default value is 0 (or disabled), which means the normal algorithm 
involves setting appropriate exclusive index-row and gap locking for 
operations. This is a difficult concept to describe in a few words. For 
a detailed description and examples refer to http://dev.mysql.com/ 
doc/refman/5.5/en/innodb-parameters.html#sysvar_innodb_locks_ 
unsafe for binlog. 


e innodb autoinc lock mode This value controls the locking mode 
that is used for generating auto increment values in InnoDB tables. 
This option supports three values: 0, which represents traditional 
mode; 1, the default, which represents consecutive mode; and 2, 
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which represents interleaved mode. A detailed description is at 
http://dev.mysql.com/doc/refman/5.5/en/innodb-auto-increment- 
handling.html. 


innodb doublewrite This variable defines if the InnoDB 
doublewrite buffer is enabled. The default is ON. This provides a 
level of crash recovery as updated data pages are first written to disk 
sequentially before they are applied in place. Setting this value to 
OFF can affect durability in a disaster recovery situation. 


innodb support xa This variable, which is enabled by default, 
provides support for an XA two-phase commit with InnoDB. If 
disabled, this can result in a different order of transactions being 
written to the binary log than the commit order. It is recommended 
that you disable this for a slave due to the single threaded nature of 
replication to improve performance. 


MySQL 5.6 Features 


In addition to some of the mentioned 5.6 variables in common replication 


sections already described, these variables are new for various 5.6 replica- 


tion features. 


Universally Unique Identifier (UUID) 


e server uuid (5.6) This is a server generated unique identifier. 
This is maintained in a separate configuration file, auto. cnf, in the 
MySQL datadir. This value is automatically generated by the 
MySQL server and should not be modified. This information is used 
by MySQL slaves to identify the master. 


More information can be found at http://dev.mysql.com/doc/refman/5.6/ 


en/replication-options.html£sysvar server uuid. 


Crash-Safe Slaves 


e master-info-repository When defined as TABLE, this variable 
will move logging of the slave log's master status from the master 
. info file to the mysql.slave master info table. 
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e relay-log-info-repository When defined as TABLE, this 
variable will move logging of the slave relay log information from the 
relay-log.info file to the mysql.slave relay log info table. 


Additional details can be found at http://dev.mysql.com/doc/refman/5.6/ 
en/replication-options-binary-log.html#option_mysqld_master-info- 
repository. 


Replication Checksums 


e binlog checksum When this option is set to CRC32 (currently 
the only possible value), the master will write a checksum for each 
event into the binary log. 


e master verify checksum When set to ON the master host will 
examine checksums that were written to the binary log when 
reading from the binary log to send events to a slave. 


e slave sql verify checksum When set to ON the slave host 
will examine and verify checksums when reading the relay log. 


Additional details can be found at http://dev.mysql.com/doc/refman/5.6/ 
en/replication-options-binary-log.html#option_mysqld_binlog-checksum. 


Multi-Threaded Slaves 


e slave parallel workers This is the number of slave worker 
threads that can be used for parallel execution of replication events 
on the slave. This requires that relay log info repository has 
a value of TABLE. 


Global Transaction Identifier (GTID) 


e gtid-mode When set to ON, this option defines that GTIDs are 
enabled on the server. GTID operations are only possible when this 
option is enabled on all master and slave servers. 


e disable-gtid-unsafe-statements When enabled, this option 
will prevent any SQL statements that cannot be logged safely in a 
transaction. This includes using non-transactional storage engine 
tables CREATE TEMPORARY TABLE and CREATE TABLE ... 
SELECT at this time. 
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Additional variables not discussed include gtid done, gtid owned, 
gtid lost,and gtid next.You can find more information on these vari- 
ables at http://dev.mysql.com/doc/refman/5.6/en/replication-options-gtids 
.html. 


User Privileges 


In addition to the available MySOL configuration variables, there are 
some replication-specific privileges that can be specified with the GRANT 
command: 


e GRANT REPLICATION SLAVE is required for retrieving binary log 
events to be applied in replication. This is the user specified in the 
CHANGE MASTER TO command. 


e GRANT REPLICATION CLIENT is required for using SHOW 
MASTER STATUS, SHOW SLAVE STATUS, and SHOW BINARY 
LOGS (statement privilege since 5.6). 


The SUPER privilege is required for SET SOL LOG BIN and SET SQL_ 
SLAVE SKIP COUNTER commands. A user that has the SUPER privilege 
will also bypass the read. only variable that is used for MySQL slave data 
integrity. 


TIP Application users should only ever have SELECT, INSERT, UPDATE, and 
DELETE on database objects in the respective application database only. Some 
other privileges for views and routines may be necessary if required; however, 
CREATE, DROP, ALTER, and SUPER should never be assigned to an 
application user accessing MySQL. That is the role of a different and separate 
DBA account with restricted host access. 


More information on the individual GRANT command options can be 
found at http://dev.mysql.com/doc/refman/5.6/en/grant.html. 


SOL Commands and Functions 


Throughout this book there have been a large number of SOL statements 
specifically for MySQL replication use in addition to the common DDL and 
DML SQL commands. These have included the following. 
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Binary Log Statements 


SHOW MASTER STATUS 

SHOW MASTER | BINARY LOGS 
SHOW BINLOG EVENTS 

PURGE MASTER | BINARY LOGS 
FLUSH BINARY LOGS 


FLUSH [MASTER] LOGS (deprecated and removed in 5.6; use 
RESET MASTER) 


RESET MASTER 
SHOW PLUGINS (5.1) 
SET SESSION SOL LOG BIN 


CAUTION While MASTER or BINARY keywords can be interchanged in 
SHOW LOGS and PURGE LOGS, FLUSH BINARY LOGS and FLUSH 
MASTER LOGS perform two very different options, the latter being very 
destructive. 


Replication Statements 


CHANGE MASTER TO 

START SLAVE [SOL THREAD |IO THREAD] [UNTIL ...] 

STOP SLAVE [SOL THREAD |IO THREAD] 

FLUSH SLAVE (deprecated and removed in 5.6; use RESET SLAVE) 
RESET SLAVE 

SHOW SLAVE STATUS 

SHOW SLAVE HOSTS 

SHOW RELAYLOG EVENTS (5.5) 

SET GLOBAL SLAVE SKIP SOL COUNTER 


For more information see http://dev.mysql.com/doc/refman/5.6/en/sql- 


syntax-replication.html and  http://dev.mysql.com/doc/refman/5.6/en/ 


show.html. 
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Replication Related Functions 


In addition, there are some replication-specific functions that can be used 
in SOL statements: 


€ MASTER POS WAITY) (since 3.23) 
e UUID() (5.6) 

e UUID SHORTY() (5.6) 

e GTID SUBSETY() (5.6) 

e GTID_SUBSTRACT() (5.6) 


For more information see http://dev.mysql.com/doc/refman/5.6/en/ 
miscellaneous-functions.html. 


Conclusion 


As stated at the beginning of the chapter, there are a large number of differ- 
ent MySQL configuration variables. The number of configurable MySQL 
variables has increased with new versions. It is important to know how a 
MySQL server has been configured and load tested in order to provide the 
best performance, reliability, and data integrity, especially with new ver- 
sions. The correct settings for your individual system can only be determined 
by understanding the load and business needs of your unique system. 
Benchmarking should be an integral part of system management and evalu- 
ation for new MySQL versions. 
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Monitoring Replication 


Monitoring MySQL is an essential requirement for a functioning pro- 
duction environment, and replication is a key component to most situations. 
Knowing if a replication server has failed is a crucial part of database admin- 
istration. A MySOL administrator should have the right tools in place to help 
identify replication issues, both proactive and reactively. In this chapter we 
will discuss the following: 

e The types of monitoring needed 

e Important information to monitor 


e Available monitoring products 
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Types of Monitoring 


Traditional monitoring tools provide common features, including a graph- 
ical interface recording historical information and providing comparative 
analysis between servers for a given timeframe. Alerting capabilities may 
be part of monitoring software or an additional product. The following sec- 
tions will discuss many of the common tools available. 


MySQL Configuration 


Any level of monitoring should also record important configuration infor- 
mation, both via the MySQL configuration file and current runtime con- 
figuration that can be modified dynamically. Monitoring a change of state 
is often overlooked and is important to provide additional information 
when understanding some observed condition. The following is an exam- 
ple script that monitors these conditions and reports any detected changes: 


$ cat check mysql config.sh 


#!/bin/sh 

# 

# Name: check mysql config 

4 Purpose: Check the current MySQL my.cnf and runtime configuration 
# for any changes. 

# Author: Effective MySQL http://effectivemysql.com 

# 


LOG _DIR="${HOME}/log" # Change appropriately 
MY CNF-"/etc/my.cnf" 


SCRIPT NAME-^basename $0 | sed -e "s/.sh$//"^ 


-z "which mysqladmin 2>/dev/null~ ] && \ 

echo "ERROR: mysqladmin not in PATH." && exit 1 

! -s "S(HOME)/.my.cnf" ] && \ 

echo "ERROR: No MySOL authentication available." && exit 1 

-z "S(LOG DIR)" ] && echo "ERROR: LOG DIR is not defined." && exit 1 

! -d "${LOG DIR)" ] && mkdir -p ${LOG FILE} 

-z "S(MY CNF]" ] && echo "ERROR: MY CNF is not defined." && exit 1 

! -s "$(MY CNF])" ] && echo "ERROR: MySQL Configuration file '$(MY CNF)' not 
found." \ 

&& exit 1 
-z "S(TMP DIR)" ] && TMP DIR-"/tmp" 





LOG FILE-"$(LOG DIR)/S(SCRIPT NAME].1log" 
CURRENT CNF-"$(LOG DIR]/my.cnf.runtime" 
CURRENT CNF FILE-"$(LOG DIR}/my.cnf.current" 


TMP_FILE="${TMP_DIR}/${SCRIPT_NAME}.tmp.$$" 
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DIFF FILE-"$(TMP DIR]/S(SCRIPT NAME].diff.$$" 


DATE TIME-^date «$Y$m$d.$H$M$S 
mysqladmin variables > $(TMP FILE] 


[ ! -s "${CURRENT_CNF}" ] && \ 

echo "S(DATE TIME} No running configuration recorded, creating." \ 
>> $(LOG FILE) && cp $(TMP FILE) ${CURRENT_CNF} 
[ ! -s "S(CURRENT CNF FILE)" ] && \ 


echo "S(DATE TIME} No my.cnf recorded, creating." >> ${LOG FILE} && \ 
cp $(MY CNF) S(CURRENT CNF FILE) 


diff -U 0 ${CURRENT_CNF FILE) ${MY_CNF} > S(DIFF FILE) 
if [ -s "$(DIFF FILE)" 
then 
echo "S(DATE TIME} WARN: A difference has been detected" >> ${LOG FILE} 
cat $(DIFF FILE) >> ${LOG FILE} 
cp $(MY CNF) S(CURRENT CNF FILE) 
CHANGES-"Y" 
fi 








diff -U 0 ${CURRENT_CNF} $(TMP FILE) > ${DIFF_FILE} 
if [ -s "$(DIFF FILE)" 
then 
CHANGES-"Y" 
echo "S(DATE TIME} WARN: A difference in config has been detected" V 
>> $(LOG FILE] 
cat $(DIFF FILE) >> ${LOG FILE} 
cp S$(TMP FILE] ${CURRENT_CNF} 


£u 
[ -z "S{CHANGES}" ] && \ 

echo "${DATE TIME} INFO: No changes detected" >> ${LOG FILE} 
#[ ! -z "S{CHANGES}" ] && \ 


# echo "Do something to alert a change has occurred" 
rm -f ${TMP_FILE} ${DIFF_FILE} 
exit 0 

This script should be modified to include an appropriate alert, for ex- 
ample, an email when a change is detected. This should be defined as a 
scheduled job run with a regular frequency. 


TIP A common mistake is to make a dynamic change to a MySQL global 
variable and not record this change in the MySQL configuration file that is 
read during a MySQL restart. 


Applying the same principle of detecting change, the MySQL error log 
and other system configuration files that can affect MySQL operations 
should also be monitored appropriately. 
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Monitoring Granularity 


In a production environment the information recorded with general moni- 
toring does not identify issues that occur with a much finer granularity. In 
many situations, replication monitoring requires a granularity of a few sec- 
onds to detect important conditions, including replication lag or failed rep- 
lication threads. For example, if replication lag was monitored every five 
minutes, then the reported value may always be zero (0) or a smaller num- 
ber. By using the example procedure defined in the appendix, lag can easily 
be more; however, if this all occurred within the five minute period there 
would be no indication. There is no maximum lag for the last five minutes 
available unless it is captured with a more real-time monitoring implemen- 
tation. Also waiting up to five minutes to know if a MySQL slave has stopped 
processing may be unacceptable. Developing a simple dashboard to display 
information is an important feature for proactive administration. 


Important MySOL Information 


This section does not contain any new information that has not already 
been discussed in previous chapters. It is important to combine informa- 
tion from various primary sources and from all MySQL instances in a giv- 
en topology to get a complete picture of an operational environment. This 
section outlines what information should be included in your applicable 
system and database monitoring. 


MySQL Error Log 


The MySQL error log will report information about the state of replication, 
including when this started, stopped, or produced an error. The following 
is a sample of the information, warning, and error messages possible: 


120709 11:32:27 [Note] Start binlog dump to master thread id(1 

Slave server(101), pos(mysql-bin.000009, 712) 

20709 :32:27 [Note] Start binlog dump to master thread id(2) 

Slave server(102), pos(mysql-bin.000009, 712) 
120709 11:32:27 [Note] Semi-sync replication initialized for transactions. 
120709 11:32:27 [Note] Semi-sync replication enabled on the master. 

20709 :32:27 [Note] Slave I/O thread: connected to master 'rsandbox@127.0.0.1 


:12630',replication started in log 'mysql-bin.000009' 
at position 712 














120709 11:32:28 [Warning] Slave SQL: If a crash happens this configuration 
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does not guarantee that the relay log info will be consistent, Error code: 0 
120709 11:32:28 [Note] Slave SQL thread initialized, starting replication in 
log 'mysql-bin.000008' at position 446, relay log './mysql_sandbox12631- 
relay-bin.000019' position: 648 
120709 11:32:28 [ERROR] Slave SQL: Error executing row event: 'Cannot 
execute statement: impossible to write to binary log since statement is in 
row format and BINLOG FORMAT - STATEMENT.', Error code: 1666 
120709 11:32:28 [Warning] Slave SQL: ... The slave coordinator and worker 
threads are stopped, possibly leaving data in inconsistent state. A restart 
should restore consistency automatically, although using non-transactional 
storage for data or info tables or DDL queries could lead to problems. In 
such cases you have to examine your data (see documentation for details). 
Error code: 1753 
Chapter 2 provides more information on the different types of error 
messages you may encounter and suitable resolution techniques. 
Detecting a change of the MySQL error log size is a basic monitoring 


step that should be implemented, regardless if you are using replication. 


SHOW MASTER STATUS 


MySQL replication requires a master with binary logging enabled. Details 
of the current binary log file and position are obtained with the SHOW 
MASTER STATUS command. Additional information on the underlying 
binary logs used and available on the master can be found with the SHOW 
BINARY LOGS command. Combined with the 1og bin and log bin 
basename (5.6) variables, the underlying filesystem files defined by these 
variables can be verified independently from any SOL statements. 


TIP  Detecting no change in the binary log size, or a change significantly greater 
or less than the average expected (generally for the given time of day) over a 
short sampling period is an additional monitoring step that can preempt a 
problem. The monitoring for a lack of volume change is just important to 
indicate a potential problem in a production system. 


SHOW SLAVE STATUS 


This SOL command is the primary source of slave replication information. 
As discussed in several chapters, the thread status via Slave IO Run- 
ningandSlave SQL Running are the most important columns of infor- 
mation to monitor. The Seconds Behind Master is also important to 
monitor if replication is lagging, or is improving or worsening over time. 
These slave status variables were detailed in Chapter 2. 
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It is important to understand that in some situations, these threads may 
not be running for a reason, for example, a backup process or a schema 
change that has stopped a thread intentionally. Applicable monitoring 
needs to account for these situations and not automatically report an error. 


SHOW GLOBAL STATUS 


Certain replication information can be obtained via SHOW GLOBAL STA- 
TUS. This includes information about binary log settings and usage, in- 
cluding replication heartbeat. Commencing in MySQL 5.5, semisynchro- 
nous replication information is obtained from SHOW GLOBAL STATUS. 


NOTE The information from SHOW GLOBAL STATUS can also be retrieved 
via the INFORMATION. SCHEMA.GLOBAL STATUS table. 


Semisynchronous Replication (5.5) 

A new feature with MySQL 5.5, semisynchronous replication can only be 
monitored via MySQL status variables. MySQL can also elect to move be- 
tween asynchronous and semisynchronous replication without any addi- 
tional notification. The status value to monitor is different between the 
master and the slave: 


master» SHOW STATUS LIKE 'Rpl semi sync master status'; 








+-------------------------------------------- +---------- + 
| Variable_name | Value | 
+-------------------------------------------- +---------- + 
| Rpl semi sync master status | oN | 
+-------------------------------------------- +---------- + 


CAUTION This is a new MySQL status variable that may not be included in 
many of the monitoring products available. 


Slave» SHOW STATUS LIKE 'Rpl semi sync slave status'; 








+-------------------------------------------- +------- + 
| Variable_name | Value | 
+-------------------------------------------- +------- + 
| Rpl semi sync slave status | ON | 
+-------------------------------------------- +------- + 


The full list of status variables on the master for semisynchronous 
replication is 


master» SHOW STATUS LIKE 'Rpl semi sync$'; 
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+-------------------------------------------- +---------- + 

Variable_name Value 
deecceeccecceReeececececeeeceeeeeeeeeeeee--e--ER-e 4--2--0------ + 

Rpl semi sync master clients 1 

Rpl semi sync master net avg wait time 321 

Rpl semi sync master net wait time 58540232 

Rpl semi sync master net waits 182303 

Rpl semi sync master no times 2 

Rpl semi sync master no tx 3106 

Rpl semi sync master status ON 

Rpl semi sync master timefunc failures 0 

Rpl semi sync master tx avg wait time 377 

Rpl semi sync master tx wait time 65273713 

Rpl semi sync master tx waits 172914 

Rpl semi sync master wait pos backtraverse 11 

Rpl semi sync master wait sessions 0 

Rpl semi sync master yes tx 182306 
4-------------------------------------------- +---------- + 


TIP Ifyou use semisynchronous replication with multiple slaves, a change of the 
Rpl semi sync master clients status variable can detect a potential 
problem within the replication topology. 


Meta Files 


MySQL includes a number of files that contain important information 
about the configuration and operation of slave replication. These are 


e datadir/master.info 


e datadir/relay log.info 


master.info 
The contents of master.info for MySQL 5.6 using the MySOL Sandbox 
example defined in the appendix are 


$ more master.info 
23 
mysql-bin.000009 
712 

127.0.0.1 

rsandbox 

rsandbox 

12630 

60 

0 
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0 
1800.000 


0 
b9a8fdel-b4a3-11e1-921d-b499baf75d68 
86400 


0 
These lines are 
. Number of lines in file 
. Current master binary log file 
. Current master binary log position 
Master host 
Master username 
Master password 


Master port 





Master connection retry time 
. SSL enabled 

. SSLCA 

. SSL CA Path 

. SSL Certificate 

. SSL Cipher 

. SSL Key 

. SSL Verify Certificate 

. Heartbeat 

. Master Bind Address 


pe 9»mugo wxo9wHMcd 


RÀ Ri oRR RH RR RH 
o N A OF PF WO N e C 


. Replicate Ignore Server IDs 


. Master UUID 


N e 
So © 


. Master Retry Count 

. SSL Certificate Revocation List (CRL) 
. SSL CRL Path 

. GTID Position enabled 


N N N 
U N e 
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This is similar to the information set with the CHANGE MASTER TO 
statement. 


relay log.info 
The contents of relay 1og.info for MySQL 5.6 using the MySQL Sand- 
box example defined in the appendix are 


$ cat relay-log.info 

6 

./mysgl sandbox12631-relay-bin.000019 
648 

mysql-bin.000008 

446 

0 


3 
3 
3 
These lines are 
Master ID 
Current relay log file 
Current relay log position 
Current master binary log 


Current master binary log position 


SQL Delay 


CD OG 9D gd & pM or 


Number of slave workers 


Meta Tables 


Starting with MySOL 5.6, important replication information can be record- 
ed in meta tables rather than underlying files. As mentioned in Chapter 3, 
the new server variables master-info-repository and relay-log- 
info-repository can be used to control the location of information. A 
value to TABLE instead of the default FILE will store replication metadata 
in two tables instead of the traditional files (master.info and relay- 
log.info).For example: 


Slave» SELECT * FROM mysql.slave_master_info\G 
kckckckckckckckckck ck ckckckck kc k k k kk kkk ]. row XX kk kk k kk KKK KKK KKK KEKE 


Master id: 2 
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Number of lines: 
Master log name: 
Master log pos: 
Host: 

User name: 

User password: 
Port: 


22 
bin-log.000005 
317 

master 

repl 
clearpassword 
3306 


Connect_retry: 10 
Enabled_ssl: 0 
Ssl_ca: 
Ssl capath: 
Ssl cert: 
Ssl cipher: 
Ssl key: 
Ssl verify server cert: 0 
Heartbeat: 1800 
Bind: 
Ignored server ids: 0 
Uuid: b1467eac-b431-11e1-8f35-001b245c2ae9 
Retry count: 86400 
Ssl crl: 
Ssl crlpath: 


Slave» SELECT * FROM mysql.slave relay log info\G 


ckckckckckckckck ck ckokck ck kokck ck k ck ck ck ck k 1, POW CECKCkckckck ck kkk ck ck okok ck ck okok ck ck k k kk 


Master id: 2 
Number of lines: 6 
/var/lib/mysql/relay-bin.000009 
273 
bin-10g.000005 
Master log pos: 317 
Sql delay: 0 
Number of workers: 0 


Relay log name: 
Relay log pos: 
Master log name: 


These tables contain information that matches the results of informa- 
tion found with SHOW MASTER STATUS and SHOW SLAVE STATUS.The 
use of tables enables easier manipulation using standard SOL commands. 


Monitoring Products 


As you saw in the previous section, the type of replication information that 
can be monitored will assist you in determining which type of monitoring 
product is the right one for your environment. A number of technologies 
listed in this section are generic monitoring products and include the abil- 
ity, either natively or with additional plugins, to monitor MySQL. An exist- 
ing monitoring product may already exist in your organization, requiring 
only the addition of MySQL specific metrics and alerts. 

The following tables list the common monitoring tools used in the in- 
dustry for MySOL monitoring. 
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Dedicated Monitoring Products 





MySQL Enterprise Monitor http://www.mysql.com/products/enterprise/monitor.html 





MySQL Performance Monitor | http://www.fromdual.ch/mysql-performance-monitor 

















Kontrolbase http://kontrollsoft.com/software-kontrollbase 
MONyog MySQL Monitor http://www.webyog.com/ 
Jet Profiler for MySOL http://www.jetprofiler.com/ 





System Monitoring Products 



































Cacti http://www.cacti.net/ 
Plugin: http://www.percona.com/downloads/percona-monitoring-plugins/ 
Nagios http://www.nagios.org/ 
Zenoss http://www.zenoss.com/ 
Plugin: http://community.zenoss.org/docs/DOC-3501 
Munin http://munin-monitoring.org/ 
Plugin: https://github.com/kjellm/munin-mysql/ 
Hyperic http://www.hyperic.com/ 
Plugin: http://support.hyperic.com/display/hyperforge/MySOL 
Ganglia http://ganglia.sourceforge.net/ 
Zabbix http://www.zabbix.com/ 
Big Brother | http://bb4.com/ 
DBTuna http://www.dbtuna.com/ 
Oracle http://oracle.com/enterprisemanager 
Enterprise Plugin: http://www.pythian.com/news/mysg|l-plugin-for-oracle-grid- 
Manager control/ 





Other Commercial System Monitoring Products 














IBM Tivoli Monitoring | http://www-01.ibm.com/software/tivoli/products/monitor/ 
Plugin: https://www-304.ibm.com/software/brandcatalog/ 
ismlibrary/details?catalog.label=1TW10TM2S 

HP Openview http://www.openview.hp.com/ 

CA Unicenter http://www.ca.com/us/infrastructure-management.aspx 











Many monitoring products require a working Linux/Apache/MySQL/ 


PHP (LAMP) stack to correctly operate. It is recommended to use a differ- 


ent system for monitoring MySQL severs. There are several client GUI de- 


velopment tools that can perform some level of real-time monitoring, for 


example, MySQL Workbench; however, this is impractical in a production 
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situation. Additional runtime graphical display commands that can be 
useful include 


e Mytop http://jeremy.zawodny.com/mysql/mytop/ 
e mtop http://mtop.sourceforge.net/ 


e InnoTop http://code.google.com/p/innotop/ 


The Implementation of Monitoring 


An important decision when choosing a monitoring product may be due to 
how information is collected. There are two implementations, using an 
agent or agentless. When a product uses an agent, a version of software is 
installed and configured on every MySQL server. When configured, the 
agent will communicate with a central repository of information that is 
used for the graphical display of information. The agent may retain infor- 
mation when the central repository is unavailable for later processing. An 
agentless monitoring product uses a centralized mechanism to poll the 
necessary MySQL servers for information and records the results at that 
time. There are advantages and disadvantages with both methods. 


MySQL Enterprise Monitor 


MySQL Enterprise Monitor (MEM) is the commercial MySOL monitoring 
product that is included with a MySOL subscription. With this solution 
you can continuously monitor your MySQL instances and be alerted to 
potential problems before a higher impact event occurs. All threshold lim- 
its within each monitor are configurable, but have a specified default value 
associated to monitoring types. This means you can toggle a threshold on 
an alert to fit into the expected behavior of your system. 


Replication Advisors Within MEM 

MEM comprises the Enterprise Dashboard web interface and the MEM 
agent for each MySQL instance. After you have installed the Enterprise serv- 
er and the agent(s) you will be able to assign advisors to a particular server 
or group of servers within the Enterprise Dashboard. The following is a list 
of replication-related advisors that are available with MEM version 2.3: 
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Binary Log File Count Exceeds Specified Limit 

Binary Log Space Exceeds Specified Limit 

INSERT ON DUPLICATE KEY UPDATE Bug May Break Replication 
Slave Detection of Network Outages Too High 

Slave Error: Unknown or Incorrect Time Zone 

Slave Execution Position Too Far Behind Read Position 

Slave Has Been Stopped 

Slave Has Experienced a Replication Error 

Slave Has Login Accounts with Inappropriate Privileges 

Slave Has Problem Communicating with Master 

Slave Has Stopped Replicating 

Slave I/O Thread Not Running 

Slave Not Configured as Read Only 

Slave Relay Log Space Is Very Large 

Slave Relay Logs Not Automatically Purged 

Slave SOL Thread Not Running 

Slave SOL Thread Reading from Older Relay Log Than I/O Thread 
Slave Too Far Behind Master 

Slave Waiting to Free Relay Log Space 

Slave Without REPLICATION SLAVE Accounts 


As you can see there are advisors that include information on perfor- 


mance, errors, lag, security, and optimal configuration. You can also devel- 


op your own custom advisors. 

In addition to these replication advisors you will be able to see all replica- 
tion topologies, what server is replicating from where, and other replication 
metadata like binary log and position. Replication topologies are automati- 
cally mapped out for you when the agent reports information to MEM. AII 
servers in a replicated set are assigned to a default group name within the 


Dashboard.The group name can be changed to something more meaningful 


in your environment. 
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Cacti 


Cacti is an open source graphing system that uses the common Round Robin 
Database (RRD) format for data storage and graphing functionality. Cacti is a 
polling-based monitoring system, not agent based like MySOL Enterprise 
Monitor. Polling a system is accomplished when the poller.php script is 
run. This script is included in the default Cacti installation. This script will 
gather SNMP and connectivity information for all of the servers configured 
within Cacti and then send out SNMP requests to those servers. The response 
back from your servers is then recorded and graphed within Cacti. This can 
lead to scaling problems depending on the quantity of servers you are moni- 
toring and the number of data points you are trying to graph. 

By default Cacti does not have alerting capabilities, but is very useful for 
visualizing the current and past state of your system. This means that you 
will be able to see system activity after it is polled within the Cacti GUI but 
will not be able to receive alerts. To harness the true power of Cacti you will 
need to use the Plugin Architecture. There are many plugins for Cacti that 
could be addressed here; however, one of the most important is thold. 
More information about Cacti can be found at http://cacti.net. 


Alerting with thold 

If you require alerting for your system the thold plugin for Cacti offers 
threshold alerts based off Cacti graphs. Adding this plugin previously re- 
quired the installation of the Plugin Architecture (PIA). Starting with Cacti 
version 0.8.8a, the PIA is included in the default Cacti code base, making it 
easier to work with all of the Cacti plugins. More information on the thold 
plugin can be found at http://docs.cacti.net/plugin:thold. This plugin has a 
prerequisite of the settings plugin to also be installed. More information 
can be found at http://docs.cacti.net/plugin:settings. 


NOTE A full list of plugins can be found at http://docs.cacti.net/plugins. 


Cacti Graph Templates 

Cacti comes with 33 default graph templates. These templates are for mon- 
itoring server characteristics like CPU, disk, memory, network, process 
count, and logged in user count. This system information is good when 
monitoring MySQL servers; however, it does not provide detailed behavior 
of mysqld itself. Graphing information specifically related to MySQL can 
be gathered by installing additional cacti graphing templates. Percona 
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provides a popular version of MySQL monitoring plugins. These are avail- 
able from http://www.percona.com/downloads/percona-monitoring-pl- 
ugins/. 

First you will need to download the latest version of the templates and 
make sure they are on your computer and also on the Cacti server. The Cacti 
server will need to access the scripts, while you will need to import the tem- 
plates through your web browser by selecting the file from your computer: 


$ wget http://www.percona.com/redir/downloads/percona-monitoring- 
plugins/percona-monitoring-plugins-1.0.1.tar.gz 
$ tar xzvf percona-monitoring-plugins-1.0.1.tar.gz 
$ cd percona-monitoring-plugins-1.0.1/ 

Notice there are two directories: one for the Cacti templates and the 
other for Nagios plugins. We are interested in the cacti directory for this 
example. 


$ ls -1 
COPYING 
Changelog 
cacti 
nagios 

There are multiple templates in the cacti directory you can install, in- 
cluding mysql, mongo, redis, and apache just to name a few. The general 
process for installation is to copy the data-gathering script into the scripts 
directory of Cacti and then import the templates via the web interface. 

On the server that hosts the Cacti installation you will need to copy the 
PHP scripts into the Cacti installation script directory. In this case Cacti has 
been installed in /var/www/html/cacti/ and the cacti scripts directory 
is /var/www/html/cacti/scripts. 


$ sudo cp percona-monitoring-plugins-1.0.1/cacti/scripts/ss get mysql stats.php \ 
/vax /www/html /cacti /scripts/ 

Now that the scripts are in place on the Cacti server you will need to log 
into the Cacti Console and import the MySOL templates by navigating to 
Console | Import Templates in your browser. In the "Import Template from 
Local File"section of the page, you can choose a file from your local computer. 
In this case we want to import the MySQL template file named cacti host 
template percona mysql server ht 0.8.6i-sver1.0.1.xml. 





NOTE Full details about how to install and configure the templates can be 
found at http://www.percona.com/doc/percona-monitoring-plugins/cacti/ 
installing-templates.html. 
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In this chapter we are most interested in monitoring replication. Con- 
tained in the newly imported templates is the Percona MySQL Replication 
GT template. The items graphed are 


e slave lag 

e Slave open tmp tbls 
e Slave rtd trnsctns 

e slave stopped 


The MySQL Replication graph displays the status of the SOL replication 
thread and replication delay. If you are using pt-heartbeat on the 
MySQL topology you will be able to use the information in the heartbeat 
table to populate this graph; otherwise, Seconds Behind Master from 
SHOW SLAVE STATUS is used. If you are using pt-heartbeat and 
would like to use that metadata to populate the graph, you will need to 
define the $heartbeat variable in the ss get mysql stats.phpscript. 


MySQL Performance Monitor (MPM) 


Created and maintained by Oli Sennhauser at FromDual, MPM is an open 
source monitoring solution based on Zabbix. This solution provides all the 
necessary modules to monitor and report MySQL performance metrics. In 
addition, MPM has preconfigured monitoring support for additional third- 
party storage engines and Galera for MySQL. FromDual also offers Moni- 
toring As A Service (MAAS) to remove the burden from existing resources. 

See http://www.fromdual.com/mysql-performance-monitor for more 
information and http://www.slideshare.net/shinguz/mysgl-monitoring- 
with-zabbix for a good presentation on usage. 


Poor Man's Replication Monitor 


An extremely inexpensive way to monitor replication would be to use a 
simple Linux shell script. Giuseppe Maxia wrote a great article on this titled 
“Refactored again: poor man's MySQL replication monitor.” See http:// 
datacharmer.blogspot.com/2011/04/refactored-again-poor-mans-mysql 
-html. The example script described in the article checks replication is run- 
ning, if the slave is replicating from the intended master, and if the slave 
host is lagging behind. If there is an error or multiple errors an email is sent 
out to a designated recipient reporting the replication issue. 
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Troubleshooting Replication Incidents 


A. good practice for administrators is to have a checklist when trying to 
determine the cause of a replication issue when an alert has been gener- 
ated. Table 8-1 shows a list that you can use as an aid when troubleshooting 
MySQL replication. 

This is not an exhaustive checklist; however, it provides a template to be 
used and enhanced accordingly for your environment and business needs. 


Step Checked What to Check? 

1 Check the MySQL error log on the master for any recent errors, either 
related to or not related to replication. 

2 Check the master to ensure log bin is enabled and verify the SHOW 
MASTER STATUS information. 

3 Ensure the master and all slave hosts have a unique server-id and 
server-uuid 

4 Ensure the master replication user has the correct privileges for 
replication. 

5 Check to see if the slave(s) are connected to the master with the SHOW 
PROCESSLIST command. 

6 Check the MySOL error log on the slave(s) for any recent errors, either 
related to or not related to replication. 

7 Check to see if the slave is running with SHOW SLAVE STATUS\G. 

8 Ensure you are using the correct master connection information on the 
slave. This is set with the CHANGE MASTER TO statement. 
Ensure you can connect to the master host from the slave using the 
mysql CLI and with telnet, for example: 
GE: 
$ mysql -urepl -psomepassword -hmaster -P3306 

9 The master information (host, user, and port) can be obtained from the 
SHOW SLAVE STATUS command. The password can be obtained from 
the master.info file, or all information can be found in the mysql . 
Slave master info table when configured. 
You can also confirm MySQL port access with telnet: 
$ telnet master 3306 
Check the firewall rules to ensure the MySQL port is not being filtered. 

10 The nmap command can be used to determine TCP/IP access to the 


master. For example: 
$ nmap -p 3306 master 


If a slave is no longer running and has an error caused by a SOL state- 
11 ment, check your data on the slave and determine if you can skip the 
error or fix the data on the slave. 


Table 8-1 Replication Checklist 
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Conclusion 


There are many ways to manually monitor MySQL replication. This chapter 
has outlined some of the tools that a database administrator can use and 
implement to provide automation of applicable monitoring.This is a practical 
and time-saving requirement for any environment, from one server to 
thousands of servers. 

Monitoring technologies can help identify performance problems and 
provide a history of information monitored for comparison and evaluation. 
You should evaluate and test some of the monitoring strategies outlined to 
determine which best fits your business. Implementing monitoring should 
be a process of continual improvement to minimize the time for regular 
maintenance operations. 

Examples and links in this chapter are available for download from 
http://EffectiveMySQL.com/book/replication-techniques. 





Appendix 
A MySQL Replication 
Test Environment 


Throughout this book, Effective MySQL: Replication Techniques in Depth, 
many examples require a working environment of a master instance and at 
least one slave instance to demonstrate the output shown. There are several 
ways to create an appropriate MySQL environment if you do not have access 
to the necessary hardware and software. The following setups are used for 
the demonstrations in this book and can be easily created to reproduce all 
the examples. Some advanced examples in this book do require additional 
operating system configuration, for example, dedicated IP addresses. 
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Manual Steps to Configure 
MySQL Replication 


It is very easy to set up MySQL replication, even on a single machine for 
testing purposes. In summary, the following steps are needed for replica- 
tion configuration between a master server and a slave server: 


e Setup two new MySQL instances. These will be represented as the 
master and the slave. 


Define binary logging with the log-bin configuration option, and a 
unique server ID with server-id on the master instance. This will 
require a MySQL instance restart. 


Obtain the master binary log position with the SHOW MASTER 
STATUS command. 


Create a new MySQL user on the master instance with host access 
from the second instance and with the REPLICATION SLAVE 
privilege. 


Define a unique server ID with server-idon the slave. This will 
require a MySQL instance restart. 


e Configure replication on the slave instance with the CHANGE 
MASTER TO command, including the details of the master instance, 
the new MySQL user, and the master instance binary log position. 


Start replication on the slave instance with the SLAVE START 
command and verify with the SHOW SLAVE STATUS command. 


The MySQL Reference Manual provides detailed instructions on how to 
configure replication at http://dev.mysql.com/doc/refman/5.5/en/replication- 
howto.html. 

When running multiple instances of MySQL on a single machine, the 
following configuration information must be different between the 
instances: 


* MySQL data directory (datadir) 
e MySQL socket file (socket) 
€ Port (port) or IP address (bind-address) 


A MySQL Replication Test Environment 245 


TIP It isacommon practice to run multiple instances of MySQL on a single 
machine by changing the MySQL communication port. Alternatively, by 
configuring multiple IP addresses on the seroer and using the bind- 
address configuration option, you can run multiple MySQL instances 
on a single server and use the default port of 3306 for all instances, which can 
simplify many commands that use the default port. 


Using MySOL Sandbox 


Created by long time MySQL community advocate and hacker Giuseppe 
Maxia, the MySQL Sandbox (http://mysqlsandbox.net/) is an essential tool 
for any database administrator. As the name implies, this tool enables you 
to create a gated sandbox of MySQL with various versions and replication 
topologies all on a single server. This open source tool is written in Perl. 
With a single command you can create the following working environ- 
ments in a few seconds: 


e Single sandboxes 

e Standard replication 

e Circular replication 

e Multiple sandboxes (same version) 

e Multiple sandboxes (different versions) 


Throughout this book we discussed several tools that can be tested and 
confirmed in a MySQL Sandbox environment. 


MySOL Sandbox Installation 


The following steps will install MySOL Sandbox on your Linux or Mac 
system: 


$ sudo cpan MySQL::Sandbox 


This step will install or upgrade MySOL Sandbox. If this is the very first 
usage of cpan on this system, you will be prompted for some default con- 
figuration that is saved for later usage. 
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Alternatively, you can install MySQL Sandbox directly from source with 
the following: 


Ur 


cd /tmp 
curl --silent -o MySQL-Sandbox.tar.gz \ 
https://launchpadlibrarian.net/91596901/MySQL-Sandbox-3.0.25.tar.gz 


Ur 


$ tar xvfz MySQL-Sandbox.tar.gz 

$ cd MySQL-Sandbox-* 

$ perl Makefile.PL 

$ make 

$ make test 

$ sudo make install 

$ make sandbox --help 

S Qd s. 

$ rm -rf MySQL-Sandbox* # Optionally cleanup temporary installation files 


CAUTION You should always refer to https://launchpad.net/mysgl-sandbox for 
the current version of MySQL Sandbox to download with the cur1 command. 


Both of these options for installation require the make command.You can 
verify this is on your system and in your current PATH with the command: 


$ which make 


If this does not return the path to the binary (for example, /Applica- 
tions/Xcode.app/Contents/Developer/usr/bin/make on MAC 
OS X), you may require additional software to be installed to use MySQL 
Sandbox. 


TIP For Mac OS X users, you will need to install Xcode from https://developer 
.apple.com/xcode or from your operating system installation DVD for either 
installation steps of MySQL Sandbox. Unfortunately, there is no simple 
command line interface to obtain and install this software without using the 
Apple App Store, which requires an account to log in, even though the software 
is free. In addition you need to install the Command Line Utilities to address 
various make complication errors. These are installed from within Xcode with 
the menu option under Preferences | Downloads | Components. 


MySQL Software Releases 


MySQL Sandbox does not include the MySQL software it installs. You can 
obtain the MySQL software to install from http://dev.mysql.com/down- 
loads for many different operating systems and architectures. The follow- 
ing steps install MySQL 5.0, 5.1 GA, 5.5 GA, and 5.6 DMR software binaries 
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for further testing using a generic Linux 64-bit software release. You should 
refer to the respective software downloads page at http://dev.mysql.com/ 


downloads for the current point release of each version. The links provided 
here are from the Effective MySOL website only to ensure a level of auto- 
mation when repeating these installation steps, because direct download 
links from the various mirror sites can easily become unavailable as newer 


point releases are released. 


$ mkdir -p $HOME/opt/mysql 
$ cd SHOME/opt/mysql 


$ curl-o mysql-5.0.95-linux-x86 64-glibc23.tar.gz 


http://effectivemysql.com/downloads/mysql-5.0.95-linux-x86 64-glibc23.tar.gz 


$ curl-o mysql-5.1.63-linux-x86 64-glibc23.tar.gz 


http://effectivemysql.com/downloads/mysql-5.1.63-linux-x86 64-glibc23.tar.gz 
$ curl -o mysql-5.5.24-linux2.6-x86 64.tar.gz 
http://effectivemysqgl.com/downloads/mysql-5.5.24-linux2.6-x86 64.tar.gz 


$ curl -o mysql-5.6.5-m8-linux2.6-x86 64.tar.gz 


http://effectivemysql.com/downloads/mysql-5.6.5-m8-linux2.6-x86 64.tar.gz 


$ for F in $HOME/opt/mysql/mysql-5*.tar.gz ; 


\ 


do make_sandbox --export_binaries $F ; done 
$ rm -rf */mysql-test */sql-bench */man */docs 


TIP For scripting purposes, add the - -silent option to the cur1 command 


to remove the interactive display of the download progress. 


The following MD5 values are provided to ensure the software matches 
original MySOL download versions: 


$ md5sum mysql*.gz 

749120f£0d9715a387b9dccb552e8c59a 
594ea37fcd9f29a9e3eddf38e7288e3f 
a9364f£4c352c92c1635b93106d56df463 
47404074376165599f8a152f£8ed08d97 


mysqi-5. 
mysqi-5. 
mysqi-5. 
mysql-5. 


0 
L 
5 
6 


.95-linux-x86 64-glibc23.tar.gz 
.63-linux-x86 64-glibc23.tar.gz 
.24-linux2.6-x86 64.tar.gz 
.5-m8-linux2.6-x86 64.tar.gz 


CAUTION You should always check the MySQL Downloads page and use the 
appropriate mirrors to obtain the current point release of each MySQL 


applicable MySQL version you wish to test. 


Replication Setup with MySOL Sandbox 


There are several different MySOL Sandbox commands to create various 
types of configurations. The following simple command will create a 


MySQL master and two slaves configuration: 


$ make replication sandbox 5.5.24 
installing and starting master 


installing slave 1 
installing slave 2 
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starting slave 1 
sandbox server started 
starting slave 2 
sandbox server started 
initializing slave 1 
initializing slave 2 
replication directory installed in $HOME/sandboxes/rsandbox 5 5 24 


This has installed a working MySQL replication topology into the 
S$HOME/sandboxes/rsandbox 5 5 24 directory. The MySQL instances 
are also started by default. A number of files and subdirectories exist here: 


$ cd S$HOME/sandboxes/rsandbox 5 5 24 
$ ls -1 
total 60 
-YrWXr-Xr-X uid gid 302 May 21 16:33 check slaves 

uid gid 465 May 21 16:33 clear all 

uid gid 785 May 21 16:33 initialize slaves 
uid gid 171 May 21 16:33 m 

uid gid 4096 May 21 17:42 master 

uid gid 4096 May 21 17:18 nodel 

uid gid 4096 May 21 20:14 node2 

uid gid 220 May 21 16:33 restart all 

uid gid 56 May 21 16:33 s1 

uid gid 56 May 21 16:33 s2 

uid gid 420 May 21 16:33 send kill all 

uid gid 612 May 21 16:33 start all 

uid gid 287 May 21 16:33 status all 

uid gid 390 May 21 16:33 stop all 

uid gid 405 May 21 16:33 use all 


-rwxr-xr-x 
-rwxr-xr-x 
-rwxr-xr-x 
drwxrwxr-x 
drwxrwxr-x 
drwxrwxr-x 
-rwxr-xr-x 
-rwxr-xr-x 
-rwxr-xr-x 
-rwxr-xr-x 
-rwxr-xr-x 
-rwXr-xr-x 
-rwxr-xr-x 


PRPPRPRPRPP RP BPP ER 


-rYWXI-Xr-X 


The MySQL instances and individual configuration can be found in the 
master, nodel, and node2 directories. All other files are convenience util- 
ities for managing the sandbox environment. The individual filenames de- 
scribe the different purposes, including starting, stopping, connecting, and 
checking the sandbox environment. 


NOTE When creating a new MySQL Sandbox replication environment 
the respective MySQL instances will be started. To stop these instances, use 
the stop a11 command found in the applicable directory. You can use the 
start allcommand to launch the MySQL instances following a shutdown 
or reboot of the host machine. 


References 


MySQL Sandbox is an open source product released under the GNU GPL 
v2. More information is available from the following sites: 
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e Product Home  http://mysqlsandbox.net/ 
e Code Download Mhttps://launchpad.net/mysql-sandbox/*download 


€ Perl CPAN package  http://search.cpan.org/ 
perldoc?MySQL::Sandbox 


e Cookbook  http://search.cpan.org/-gmax/MySQL-Sandbox-3.0.25/ 
lib/MySQL/Sandbox/Recipes.pm 


Using Virtual Servers 


There are many different virtualization products available that can support 
running a virtual environment on a single server. The following steps show 
how to set up a dedicated MySQL server environment using VirtualBox. 


VirtualBox Installation 


VirtualBox is an open source product that can be downloaded from https:// 
www.virtualbox.org/. This runs on multiple operating systems, including 
Mac OS X, Microsoft Windows, Linux, and Solaris. Refer to the website for 
specific instructions on installation with your operating system at http:// 
www.virtualbox.org/manual/ch02.html. 

Following the installation of the virtualization tool on your machine, 
referred to as the host, you also need to install an operating system to oper- 
ate a virtual host. This is also known as a guest. For this demonstration we 
will be using the Ubuntu Server 12.04 LTS 64-bit operating system available 
from http://www.ubuntu.com/download/server. Detailed instructions for 
configuration of a virtual server can be found at https://www.virtualbox 
.org/manual/ch03.html. 

As part of the installation process you will be required to specify a 
username. There is no restriction to what this must be. For demonstration 
purposes in this book, the username will be^user." 


VirtualBox Configuration 

After installing and configuring a new virtual host, one setting is necessary 
for use in the examples throughout this book. Under the settings for the 
virtual host, select Network and enable Adapter 2, as shown in Figure A-1. 
This should be configured to attach to a bridged adapter. The name should 
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Figure A-1 Virtual Box Network Adapter 2 configuration 


also match your applicable network device on your host. Adapter 1 should 
remain enabled as a NAT Adapter. 


MySQL Replication Environment 
For this book, the following environment of three servers is used: 


e Server 1 is called alpha, and has an IP address of 192.168.1.51 
e Server 2 is called beta, and has an IP address of 192.168.1.52 
e Server 3 is called gamma, and has an IP address of 192.168.1.53 


The IP addresses can be defined at your discretion. It is recommended 
you use the same C class as your host machine and internal network, i.e., 
the 192.168.1. portion should be configured to match your own network. 
The following configuration is repeated for each new VirtualBox guest host 
that is created. VirtualBox also enables you to easily create one guest host 
and use the clone option. 
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Virtual Host Configuration 
The following steps are necessary to configure the default Ubuntu Server 
installation to match the needs for the examples. 


Sudo Privileges 
To automate many commands in the examples for this book, removing the 
password requirements when executing the sudo command will remove un- 
necessary complexity. The following steps will grant the user this privilege: 
echo ^id -un^" ALL-NOPASSWD: ALL" > /tmp/mysql 
chmod 440 /tmp/mysql 


$ 

$ 

$ sudo chown root:root /tmp/mysql 

$ sudo mv /tmp/mysql /etc/sudoers.d 


Hostname 
For each instance we set the hostname by changing this in the /etc/host- 
name file: 


$ sudo vi /etc/hostname 
The contents of this file should look like: 


$ cat /etc/hostname 
alpha 


The contents of this file should reflect alpha, beta, and gamma, 
respectively. 


DNS Hosts 
In order to refer to the server hostnames in all examples, the following 
values are added to each virtual server hosts file: 


$ sudo /etc/hosts 
The contents of this file should look like this for all virtual servers: 


$ cat /etc/hosts 
127.070. 1 localhost 


192.168.1.51 alpha 
192.168.1.52 beta 
192.168.1.53 gamma 


In addition, the default installation will create an entry for the default 
hostname at 127.0.[01].1. This line in the hosts file must be removed. 
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TIP Ifyou use your host machine to execute any commands, it is highly 
recommended you add these entries to the host machine's respective hosts file. 
This will vary depending on your operating system. 


Server IP Address 

In order to be able to reference hosts in a reproducible fashion, each vir- 
tual server must be defined with a static IP address rather than a dynamic 
DHCP IP address. This is specified in the network interfaces file by adding 
entries for eth0 and eth1: 


$ sudo vi /etc/network/interfaces 
The contents of this file should look like: 


$ cat /etc/network/interfaces 

# The loopback network interface 
auto lo 

iface lo inet loopback 


# Adapter 1 
auto etho 
iface ethO inet dhcp 


# Adapter 2 

auto ethl 

iface ethl inet static 
address 192.168.1.51 
netmask 255.255.255.0 
network 192.168.1.1 
broadcast 192.168.1.255 


The address value should represent the corresponding value for each 
virtual server. All other information remains unchanged. 


NOTE Ifyou clone a guest virtual server to create a new virtual server, be sure 
to select the option to create a new MAC address for each network device. 


The preceding network definition relies on the virtual network adapter 
name in the guest server to be eth0 and eth1.These are the default values 
for the first two network adapters, and when you create your first guest 
instance this should operate without incident. 


Adjusting Cloned Virtual Hosts 
If you choose to clone your first guest virtual server to avoid repeating 
these steps, it is likely the virtual server will not correctly define etno 
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and eth1.This is due to the change of MAC address, which is necessary to 
operate multiple virtual hosts concurrently. To address this problem, you 
must correctly reference the MAC address to the virtual adapter name in 
the /etc/udev/rules.d/70-persistent-net.rules file. Generally, 
this involves removing older entries that match etho and eth1, and up- 
dating the following entries with the correct name. For reference purposes, 
the format of this file follows. The ATTR (address) value should match the 
MAC address defined in the Network tab of your guest virtual server for 
Adapter 1 (NAT) and Adapter 2 (Bridged Adapter), respectively. The NAME 
value should match etno and eth1, respectively: 

$ cat /etc/udev/rules.d/70-persistent-net.rules 

# PCI device 0x8086:/sys/devices/pci0000:00/0000:00:03.0 (e1000) 
SUBSYSTEM--"net", ACTION--"add", DRIVERS=="?*", 
ATTR{address}=="08:00:27:58:24:de", ATTR{dev_id}=="0x0", 
ATTR{type}=="1", 

KERNEL--"eth*", NAME="eth0o" 

# PCI device 0x8086:/sys/devices/pci0000:00/0000:00:08.0 (e1000) 
SUBSYSTEM--"net", ACTION=="add", DRIVERS=="?*", 
ATTR{address}=="08:00:27:d4:5£:91", ATTR{dev_id}=="0x0", 
ATTR{type}=="1", 

KERNEL--"eth*", NAME-"ethl" 








CAUTION If you clone your guest virtual server, it is likely that internally the 
virtual adapter name will not match expectations and networking will not be 
correctly configured. A manual change to the mapping on the guest operating 
system will be required. 


Additional network reference information can be found at https://help 
-ubuntu.com/12.04/serverguide/network-configuration.html. 


MySQL Installation 

The following steps will manually install MySQL 5.6 on each virtual server 
that has just been created. Any MySQL version can be installed by chang- 
ing the appropriate value on the first line to a matching source MySOL 
binary file as described in the MySOL Sandbox section: 

MYSQL TAR-"mysql-5.6.5-m8-linux2.6-x86 64.tar.gz" 

som apt-get update 

sudo apt-get install libaiol # This is only needed for MySQL 5.6 
rm -f S$HOME/.my.cnf 


wget http://effectivemysql .com/downloads/${MYSQL TAR} 
md5sum ${MYSQL TAR} 


VoU Ur Xr UY UY UY 
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tar xvfz $[MYSOL TAR} 

rm -f $(MYSOL TAR} 

mv mysql-5* mysql 

cd mysql 

./scripts/mysql install db 
./bin/mysqld safe & 

sleep 3 

tail data/~hostname~.err 


Ur Ur UY UY XY UY UY UY 


The output of the MySQL error log should be verified to confirm that 
MySQL successfully started: 


120615 12:32:41 [Note] /home/user/mysql/bin/mysqld: ready for connections. 
Version: '5.6.5-m8' socket: '/tmp/mysql.sock' port: 3306 MySQL Community 
Server (GPL) 


A second sanity check can also be performed: 


$ ./bin/mysgl -e "SELECT VERSION()" 


as Sa SS + 
| VERSION() | 
Poses eS SSeS + 
| 5.6.5-m8 | 
foes SSS + 


NOTE For demonstration purposes MySQL installation is performed using the 
applicable binary tar to easily support multiple MySQL versions. The software 
is installed in the home directory of the current user for simplicity only. These 
steps are for testing purposes only and do not reflect a suitable practice for a 
production deployment. 


The following step will help to secure the MySQL installation. During 
this step you are asked to specify a root password for the MySQL instance. 
This value is also used in a subsequent configuration file. For demonstra- 
tion purposes this value is” passwd”; however, any value can be specified if 
configured appropriately in the subsequent step: 


$ ./bin/mysql secure installation 

# Follow the prompts 

$ ./bin/mysql -uroot 

ERROR 1045 (28000): Access denied for user 'root'G'localhost' 
(using password: NO) 

$ HOSTNAME- hostname 

$ echo "[client] 

user-root 

password-passwd 

[mysql] 
prompt='${HOSTNAME}> '" > SHOME/.my.cnf 
$ ./bin/mysql -e "SELECT VERSION ()" 
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NOTE For demonstration purposes in this book the root MySQL user is used, 
and a configuration file is created for simplicity of password authentication. In 
a production environment, appropriate procedures to match your business 
needs should be implemented accordingly. 


The final steps help in command pathing and basic configuration to en- 
able tools and replication configuration happen unaided: 


$ echo "export MYSQL HOME=\SHOME/mysql 

export PATH=\S$MYSQL HOME/bin:\S$PATH" > /tmp/mysql.sh 
sudo mv /tmp/mysql.sh /etc/profile.d/ 

. /etc/profile.d/mysql.sh 

$ mysql -e "GRANT ALL ON *.* TO root@'192.168.1.%' \ 
IDENTIFIED BY 'passwd' WITH GRANT OPTION" 

cd $HOME/mysql 

mkdir etc 

SERVER ID-^ grep address /etc/network/interfaces | cut -d. -f4^ 
echo " [mysqld] 

server-id=${SERVER_ID} 

log-bin" > etc/my.cnf 

$ mysqladmin shutdown 

$ mysqld safe --defaults-file-$HOME/mysql/etc/my.cnf & 
$ sleep 2 

$ tail data/~hostname~.err 





UY Xr Ur Ur 


These steps have not completed the replication setup. Each virtual server 
has MySQL running and configured to support being used in a MySOL 
replication topology. The MySQL utilities described in Chapter 4 are used 
to complete the necessary steps. Alternatively, the manual instructions de- 
fined in this appendix may be used to complete the setup. 


Testing and Verifying MySQL Replication 


The following simple test can be used to show MySQL replication (and lag) 
in operation. Also from Giuseppe, the following simple procedure produces 
a repeating and increased statement duration that demonstrates replication 
lag using the default MySOL asynchronous replication. This requires two 
separate terminal sessions. 

In one terminal session perform the following command to monitor 
MySQL replication in action using the MySOL Sandbox setup previously 
described: 


$ cd S$HOME/sandboxes/rsandbox 5 5 24 

$ ./restart all 

$ while [ : ] ; do date; ./sl -e "SHOW SLAVE STATUSNG" | \ 
grep Seconds; sleep 1; done 
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In a different terminal session run the following commands: 


$ cd SHOME/sandboxes/rsandbox 5 5 24 
$ ./m 
dhttp://datacharmer.blogspot.com/2006/06/filling-test-tables-quickly.html 
CREATE SCHEMA IF NOT EXISTS book3; 
USE book3 
DROP TABLE IF EXISTS numbers; 
CREATE TABLE numbers (id INT NOT NULL PRIMARY KEY); 
DELIMITER $$ 
DROP PROCEDURE IF EXISTS fill numbers $$ 
CREATE PROCEDURE fill numbers () 
DETERMINISTIC 
BEGIN 
DECLARE counter INT DEFAULT 1; 
TRUNCATE TABLE numbers; 
INSERT INTO numbers VALUES (1); 
WHILE counter « 1000000 


DO 
NSERT INTO numbers (id) 
SELECT id + counter 
FROM numbers; 
SELECT COUNT(*) INTO counter FROM numbers; 
SELECT counter; 
END WHILE; 
END $s 
DELIMITER ; 
CALL fill numbers () ; 





This SQL statement will produce the following output. Depending on 
your hardware this may take 20 seconds or longer to complete: 


1 row in set (1.11 sec) 


1 row in set (2.09 sec) 
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1 row in set (4.10 sec) 








1 row in set (7.83 sec) 


The result from the initial monitoring terminal shows MySQL in opera- 
tion and that replication lag is present: 


Sun May 20 20:11:06 EDT 2012 
Seconds Behind Master: O0 
Sun May 20 20:11:07 EDT 2012 
Seconds Behind Master: 1 
Sun May 20 20:11:08 EDT 2012 
Seconds Behind Master: 1 
Sun May 20 20:11:09 EDT 2012 
Seconds Behind Master: 2 
Sun May 20 20:11:10 EDT 2012 
Seconds Behind Master: 3 
Sun May 20 20:11:11 EDT 2012 
Seconds Behind Master: 4 
Sun May 20 20:11:12 EDT 2012 
Seconds Behind Master: 5 
Sun May 20 20:11:13 EDT 2012 
Seconds Behind Master: O0 
Sun May 20 20:11:14 EDT 2012 
Seconds Behind Master: 6 
Sun May 20 20:11:16 EDT 2012 
Seconds Behind Master: 8 
Sun May 20 20:11:17 EDT 2012 
Seconds Behind Master: 9 
Sun May 20 20:11:18 EDT 2012 
Seconds Behind Master: 10 
Sun May 20 20:11:19 EDT 2012 
Seconds Behind Master: 0 








You can also monitor MySQL replication in real time with the SHOW 
SLAVE STATUS command.The following watch syntax provides an online 
display that does not translate into print: 
$ cd S$HOME/sandboxes/rsandbox 5 5 24 


$ watch --interval-1 --differences \ 
'mysql --defaults-file-nodel/my.sandbox.cnf -e "SHOW SLAVE STATUS VNG"' 
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When using MySQL Sandbox you can simplify this syntax using the 
convenience command for connecting to the slave: 


$ watch --interval-1 --differences  './s1 -e "SHOW SLAVE STATUS NG"' 


MySQL Sandbox also provides a quick command for viewing and mon- 
itoring a subset of replication information: 


$ ./check slaves 
Slave #1 
Master Log File: mysql-bin.000002 
Read Master Log Pos: 5308 
Slave IO Running: Yes 
Slave SQL Running: Yes 
Exec Master Log Pos: 5059 
slave # 2 
Master Log File: mysql-bin.000002 
Read Master Log Pos: 5308 
Slave IO Running: Yes 
Slave SOL Running: Yes 
Exec Master Log Pos: 5059 








NOTE The true replication delay is not Seconds Behind Master. This 
value is an indication of the current system time with the time the current 
statement was executed on the master. It is possible that no statements have 
since physically occurred, and replication will be in sync with the completion of 
the running statement, which may be a few seconds, when a value of hundreds 
is reported. This value is a good indication that the slave has not yet completed 
executing all available statements in the replication stream. 


Conclusion 


These are just two simple examples of how to create a suitable MySOL 
replication environment for testing purposes. 

Examples and links in this chapter are available for download from 
http://EffectiveMySQL.com/book/replication-techniques. 
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slave servers, 14-15 
crash-safe, 58-60 
multi-master replication, 94-100 
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